Audio and video program recording, editing and playback systems using metadata

ABSTRACT

A system for utilizing metadata created either at a central location for shared use by connected users, or at each individual user&#39;s location, to enhance user&#39;s enjoyment of available broadcast programming content. A variety of mechanisms are employed for automatically and manually identifying and designating programming segments, associating descriptive metadata which the identified segments, distributing the metadata for use at client locations, and using the supplied metadata to selectively record and playback desired programming.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of the filing date of thefollowing co-pending applications: U.S. Provisional Patent ApplicationSerial No. 60/297,204 filed on Jun. 8, 2001 entitled “Methods andApparatus for Navigating Time-Shifted Television Programming;” U.S.Provisional Patent Application Serial No. 60/352,788 filed Nov. 28, 2001entitled “Methods and Apparatus for Distributing Segmented TelevisionProgramming;” U.S. Utility patent application Ser. No. 09/536,969 filedon Mar. 28, 2000 entitled “Systems and Methods for Modifying BroadcastProgramming;” U.S. Provisional Application Serial No. 60/304,570 filedon Jul. 11, 2001 entitled “Audio and Video Program Recording, Editingand Playback Systems using Metadata;” U.S. Provisional ApplicationSerial No. 60/336,602 filed on Dec. 3, 2001 entitled “Methods andApparatus for Automatically Bookmarking Programming Content;” and U.S.Utility application Ser. No. 10/060,001 filed on Jan. 29, 2002 entitled“Audio and Video Program Recording, Editing and Playback Systems UsingMetadata.” The disclosure of each of the foregoing applications ishereby incorporated herein by reference.

REFERENCE TO COMPUTER PROGRAM LISTING APPENDIX

[0002] A computer program listing appendix is stored on each of twoduplicate compact disks which accompany this specification. Each diskcontains computer program listings that illustrate implementations ofthe invention. The listings are recorded as ASCII text in IBM PC/MS DOScompatible files which have the names, sizes (in bytes) and creationdates listed below: Size (bytes) Date created Filename  3,123 May 21,2001 12:08 p ToolBarDlg.h  1,141 May 21, 2001 12:08 p CAnimate.h  65,419Jun. 05, 2001 11:13 p CNPDlg.cpp  5,228 Jun. 04, 2001 10:20 a CNPDlg.h 33,964 Jun. 04, 2001 10:16 a ConceptDlg.cpp  6,337 Jun. 01, 2001 10:30a ConceptDlg.h  19,459 Jun. 04, 2001  5:37 p CPLDlg.cpp  3,553 May 23,2001  4:24 a CPLDlg.h  3,758 May 21, 2001 12:08 p cpp  26,672 Jun. 04,2001  5:39 p CSegmentDriver.cpp  6,317 May 29, 2001  2:04 pCSegmentDriver.h  29,715 May 21, 2001 12:08 p dxmplayer.cpp  7,527 May21, 2001 12:08 p dxmplayer.h  1,202 May 21, 2001 12:08 pDXMPLayerConstants.h  5,258 May 21, 2001 12:08 p DXMPlayerEventSink.cpp 2,239 May 21, 2001 12:08 p DXMPlayerEventSink.h  7,753 Jun. 01, 200110:28 a GClient.clw  3,259 May 24, 2001 12:14 p GClient.cpp  5,771 May21, 2001 12:08 p GClient.dep  13,602 Jun. 04, 2001 10:21 a GClient.dsp   537 May 21, 2001 12:08 p GClient.dsw  10,333 Jun. 05, 2001  1:09 pGClient.h  16,693 May 21, 2001 12:08 p GClient.mak 541,696 Jun. 01, 200110:24 a GClient.ncb  79,872 Jun. 01, 2001 10:23 a GClient.opt  1,704Jun. 04, 2001 10:26 a GClient.plg  35,194 Jun. 05, 2001 10:46 pGClient.rc  4,127 May 21, 2001 12:08 p GClientDlg.cpp  1,353 May 21,2001 12:08 p GClientDlg.h  12,074 May 21, 2001 12:08 p gmanager.cpp 3,404 May 21, 2001 12:08 p gmanager.h  7,706 May 23, 2001  4:19 aIMediaPlayer.h  20,350 Jun. 04, 2001  3:56 p InfoDlg.cpp  3,187 May 24,2001 12:17 p InfoDlg.h  8,988 May 21, 2001 12:08 p ISegmentDriver.h 3,020 May 28, 2001  6:52 a ISegmentListCtrl.h  10,152 May 23, 2001 4:22 a MediaPlayer.cpp  1,894 May 23, 2001  4:19 a MediaPlayer.h  5,982Jun. 04, 2001 10:13 a metadata.cpp  33,088 Jun. 05, 2001  1:13 pMetaDataDlg.cpp  3,754 Jun. 01, 2001 10:21 a MetaDataDlg.h  13,369 Jun.05, 2001  1:31 a NotAuthored.cpp  3,292 May 24, 2001  6:31 aNotAuthored.h  11,161 Jun. 05, 2001  3:53 p playlist.cpp  5,676 May 24,2001  4:20 a ReadMe.txt  13,465 Jun. 04, 2001 10:19 a Resource.h  3,001May 21, 2001 12:08 p resource.h.ejb  2,622 May 21, 2001 12:08 presource.h.old  6,716 Jun. 04, 2001  3:53 p segment.cpp  3,608 May 31,2001  1:54 p segment.h  1,227 May 21, 2001 12:08 p segmentList.cpp 1,598 May 21, 2001 12:08 p segmentList.h  42,155 Jun. 05, 2001  1:13 pSegmentListCtrl.cpp  8,025 May 28, 2001  6:52 a SegmentListCtrl.h 21,934 May 21, 2001 12:08 p segmentlistctrl.sav  19,437 Jun. 05, 200110:02 a SettingsDlg.cpp  3,086 May 28, 2001  6:56 a SettingsDlg.h    209May 21, 2001 12:08 p StdAfx.cpp  1,195 May 21, 2001 12:08 p StdAfx.h 8,653 Jun. 05, 2001  1:33 a ThankYouDlg.cpp  2,288 Jun. 04, 2001 10:08a ThankYouDlg.h  11,679 Jun. 04, 2001 10:25 a ToolBarDlg.cpp    662 May21, 2001 12:07 p CAnimate.cpp  5,993 Apr. 19, 2001  3:39 pTestAccess.java  3,060 Mar. 27, 2001  5:16 p ER.TVP  2,043 Apr. 19, 2001 3:55 p GDBPool.java.txt  7,307 Apr. 23, 2001 12:16 p GGUPIServlet.java 1,472 Apr. 27, 2001  5:23 p GPlayListSet.java.txt  1,189 Apr. 27, 2001 5:26 p GPlayListSetFromCache.java.txt  1,207 Apr. 27, 2001  5:28 pGPlayListSetFromDatabase.java.txt  1,187 Apr. 27, 2001  5:27 pGPlayListSetFromFile.java.txt  7,337 Apr. 23, 2001 12:08 pGPostServlet.java  3,592 Apr. 27, 2001  5:16 p GQuery.java.txt  7,308Apr. 23, 2001 12:16 p GQueryServlet.java  1,643 Apr. 19, 2001  5:05 pGService.java.txt    825 Apr. 12, 2001 12:24 a GTest.java  1,132 Apr.20, 2001  2:00 p JDBCConfig.properties  5,604 Apr. 23, 2001 11:51 aSnoopServlet.java  5,874 Apr. 19, 2001 12:10 a Copy ofTestAccess.java.txt    935 May 29, 2001  1:01 p tvp.php    358 May 25,2001  7:30 a frames.html  12,051 Jun. 05, 2001  3:25 p gspot.php    919May 22, 2001  6:37 a movie2.php    636 May 22, 2001  6:37 a player.js 4,546 Jun. 01, 2001  4:23 p sreplace.php    983 May 31, 2001  6:04 afile.ph

COPYRIGHT STATEMENT

[0003] A portion of the disclosure of this patent document containsmaterial which is subject to copyright protection. The copyright ownerhas no objection to the facsimile reproduction by anyone of the patentdocument or the patent disclosure, as it appears in the Patent andTrademark Office patent file or records, but otherwise reserves allcopyright rights whatsoever.

FIELD OF THE INVENTION

[0004] This invention relates to audio and video program reception,storage, editing, recording and playback systems and more particularlyto methods and apparatus for distributing, recording, organizing andediting metadata that is used to selectively distribute, record,organize, edit and play program content.

BACKGROUND OF THE INVENTION

[0005] Historically, the viewing experience of TV has been governed bythe content and programming of the service providers, broadcasters andnetworks who decide when programs will be available and their duration.While lifestyles have become more complex and the content available tothe viewer has increased, it has become more desirable to allow viewersto control this form of entertainment on their own terms. While videocassette recorders (VCRs) allow viewers to capture content for futureplayback, the VCR has been plagued with limitations inherent in theanalog tape media and the difficulty viewers commonly experience inprogramming these devices to record selected future programs.

[0006] The recent advent of digital video recorders (DVRs), coupled withmore intuitive electronic program guides (EPGs) used in popular DVRs,have provided new and simplified recording options for viewers. Inaddition, as a useful byproduct of digital storage, DVRs provide theability to pause, replay and fast-forward the playback of time-shiftedprogramming. However, as the number of available channels and the volumeand diversity of available content increases, currently available DVRsand program guides will not provide the needed ability to playback andscan volumes of stored video with simple controls and with minimalknowledge of the available content. Viewers will need “more informationabout the content to help them navigate between programs and within aparticular program.

SUMMARY OF THE INVENTION

[0007] In a principle aspect, the present invention takes the form ofmethods and apparatus for selectively reproducing recorded video programsegments retrieved from a mass storage device under the control ofplaylist metadata which identifies a selected set of the stored segmentsant the ordered sequence in which those segments are to be reproduced inthe absence of an intervening control command from the viewer. Theplaylist metadata includes a text description of each segment in thesequence. In response to a request from the viewer, an segment guidelisting containing the text description of each segment is displayedwith the text description of the currently playing segment beingvisually identified on the list. Control means operated by the viewerpermit the viewer to choose a different segment to be viewed byselecting the text description of that segment on the displayed indexlisting.

[0008] In accordance with a further feature of the invention, attributedata is associated with at least selected ones of the stored videoprogram segments and means are provided for selecting and sorting storedsegments based on the attribute data.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009]FIG. 1 is a schematic block diagram that illustrates thefunctional components which are used in a preferred embodiment ofinvention and which operate at both a remote location and at one of theuser locations to implement the invention;

[0010]FIG. 2 is a data flow diagram illustrating the manner in whichvideo program content and descriptive metadata is transferred betweencontent and service providers and a personal video recording deviceoperated by a viewer in accordance with the invention; and

[0011] FIGS. 3-5 illustrate screen layout displays illustrating themanner in which program segment guides are displayed to enable the userto interactively control program segment playback as defined by playlistmetadata.

DETAILED DESCRIPTION

[0012] Background

[0013] The present invention belongs to a family of related systems thatuse metadata to control the playback of broadcast programming asdisclosed in the previously issued patents and previously filedapplications summarized below.

[0014] U.S. Pat. Nos. 5,892,536 and 5,986,692, issued to James D. Loganet al. describe systems which employ metadata to selectively store,manipulate and playback broadcast programming. Some of the novelarrangements and features disclosed in those two patents may besummarized as follows:

[0015] 1. A remote editing station, which may be at the broadcastfacility or at a remote location, classifies, describes or otherwiseidentifies individual segments of broadcast programming and sendsmetadata (sometimes referred to as “markup data”) identifying anddescribing those segments to a remote client receiver. For example, themarkup data may identify individual segments by specifying the sourceand the time of the original broadcast, or by specifying some otherunique characteristic of the broadcast signal. The program segments maybe TV, radio, or Internet programs, or portions of programs, includingindividual songs, advertisements, or scenes.

[0016] 2. The communication link used to transmit the metadata to theclient may take one of several forms, including the Internet, a dialuptelephone link, the communications pathway used to carry the broadcastsignals to the client, or other forms of communication used to transportthe metadata to the client.

[0017] 3. At the client receiver, the metadata is used to identifyparticular program segments that may then be manipulated in one or moreof a variety of ways. For example, the metadata may be used toselectively play back or record particular segments desired by the user;to re-sequence the identified segments into a different time order; to“edit-out” undesired portions of identified segments; to splice newinformation, such as computer text or advertising, into identifiedsegments for rendering with the program materials, or to substitutedifferent material (e.g. dubbing in acceptable audio to replaceprofanity to make programming more acceptable to minors).

[0018] 4. The client receives and locally stores incoming broadcastprogramming and uses the markup data to identify desired segments withinthe stored program materials. The local storage mechanism mayadvantageously include means for concurrently recording livebroadcasting while replaying a delayed version of the previouslyrecorded programming as described in U.S. Reissue Patent 36,801 issuedto James D. Logan et al.

[0019] 5. The markup data can provide a detailed “electronic programguide” to the broadcast programming previously received and stored in apersonal video recorder (PVR) or an audio storage device, permitting theuser to selectively play back a desired segment or portion of theprogramming previously recorded.

[0020] 6. The markup data may be used to create a recorded collection ofdesired segments extracted from the buffered broadcast, allowing thedesired segments to be saved while the remainder of the bufferedmaterials is discarded to conserve recording space.

[0021] 7. Special markup signals may be selectively sent to individualsubscribers based on his or her indicated preferences so that onlypreferred program segments are identified and processed. For example, asubscriber might request markup data only for sports and news.

[0022] U.S. Pat. No. 6,088,455 issued to James D. Logan et al. describesrelated systems that use a signal analyzer to extract identificationsignals from broadcast program segments. These identification signalsare then sent as metadata to the client where they are compared with thereceived broadcast signal to identify desired program segments. Forexample, a user may specify that she likes Frank Sinatra, in which caseshe is provided with identification signals extracted from Sinatra'srecordings which may be compared with the incoming broadcast programmingcontent to identify the desired Sinatra music, which is then saved forplayback when desired.

[0023] U.S. patent application Ser. No. 09/536,696 filed by James D.Logan et al. on Mar. 28, 2000 describes further systems that employmetadata for selectively recording and/or reproducing received broadcastprogramming. The implementations disclosed in that application employ:

[0024] 1. A receiver connected to record incoming broadcast signals anda PC connected to a web server via the Internet. A browser programrunning on the PC uses the web interface provided by the web server,selects songs of interest, downloads identification signals (e.g.,extracted feature-sets or signatures) which uniquely identify thecontent of desired program segments (songs), which are then selectivelysaved for reproduction.

[0025] 2. A signal processor that identifies characteristics of thestored programming (scene changes, voice vs. music, voices of particularpeople, etc.) that can be used to selectively store desired programming.

[0026] 3. Identification signals derived from received broadcastprogramming at the client produce identification signals which are sentto a remote server which compares the received identification signalswith a database at the server and returns attribute information to theclient to describe recognized information. The attribute information caninclude the title of the segment, the name of the performing artist,albums that have a recording of this segment, etc.

[0027] 4. Program segment files (e.g. songs) in a server library thatare made available to those client locations which demonstrate that theyare entitled to access the library copy by sending an identificationsignal to the server that is extracted from a copy of the desiredsegment already in the client's possession. Thereafter, a qualifiedclient can obtain the authorized copy from the server from remotelocations. Locally recorded programming can be uploaded from a clientinto the library, and such uploading can be “virtual” (that is, need notactually take place) when an equivalent copy of the same program segmentis already stored in the server library.

[0028] U.S. Pat. Nos. 5,271,811, 5,732,216, and 6,199,076, andco-pending application Ser. No. 09/782,546 filed on Feb. 13, 2001, byJames D. Logan et al. describe an audio program and message distributionsystem which incorporates the following features:

[0029] 1. A host system organizes and transmits program segments toclient subscriber locations.

[0030] 2. A scheduling file of metadata schedules the content andsequence of a playback session, which may then be modified by the user.

[0031] 3. The content of the scheduled programming is varied inaccordance with preferences associated with each subscriber.

[0032] 4. Program segments are associated with descriptive subjectmatter segments, and the subject matter segments may be used to generateboth text and audio cataloging presentations to enable the user to moreeasily identify and select desirable programming.

[0033] 5. A playback unit at the subscriber location reproduces theprogram segments received from the host and includes mechanisms forinteractively navigating among the program segments.

[0034] 6. A usage log is compiled to record the subscriber's use of theavailable program materials, to return data to the host for billing, toadaptively modify the subscriber's preferences based on actual usage,and to send subscriber-generated comments and requests to the host forprocessing.

[0035] 7. Voice input and control mechanisms included in the playerallow the user to perform hands-free navigation of the program materialsand to dictate comments and messages, which are returned to the host forretransmission to other subscribers.

[0036] 8. The program segments sent to each subscriber may includeadvertising materials, which the user can selectively play to obtaincredits against the subscriber fee.

[0037] 9. Parallel audio and text transcript files for at least selectedprogramming enable subject matter searching and synchronization of theaudio and text files.

[0038] 10. Speech synthesis may be used to convert transcript files intoaudio format.

[0039] 11. Image files may also be transmitted from the server forsynchronized playback with the audio programming. 12. A text transcriptincluding embedded markup flags may be used to provide a programmedmultimedia presentation including spoken audio text created by speechsynthesis synchronized with presentation of images identified by themarkup tags.

[0040] U.S. Utility application Ser. No. 10/060,001 filed on Jan. 29,2002 entitled “Audio and Video Program Recording, Editing and PlaybackSystems Using Metadata” describes means at the user's location forcreating metadata which may be used in combination with metadataprovided by an external source, for editing metadata in various ways atthe user's location, for automatically responding to user activity togenerate new metadata which characterizes the user's preferences andwhich serves to automatically identify and describe (or rate)programming segments, and for responding in novel ways to the availablemetadata to enhance the utility and enjoyment of available broadcastmaterials. Methods and apparatus are employed for selectivelycontrolling the presentation of broadcast programming in which a userviewing or listening to broadcast programming at a first location maytake advantage of the insights provided by a different viewer at anotherlocation in order to control the manner in which segments of thebroadcast programming are recorded and/or replayed.

[0041] The disclosure of each of the foregoing patents and applicationsis incorporated herein by reference.

[0042] Architectural Overview

[0043] The methods and apparatus contemplated by the present inventionfacilitate the selective storage, organization and reproduction(playback) of broadcast programming through the use of metadata thatidentifies and describes segments of that broadcast programming. Thismetadata can be created locally or at a remote site and transmitted tothe user's location to enable the user to more effectively managebroadcast programming received at the user's location.

[0044]FIG. 1 illustrates in schematic form the manner in whichinformation is processed in accordance with the invention. As will bedescribed in more detail below, many of the structures and functionsillustrated in FIG. 1 represent alternative or optional functionalitythat need not be present in all implementations of the invention.

[0045] At the remote location, broadcast programming from a source 100is received at 101 and may be processed immediately or saved in astorage unit 103 for later processing. The incoming broadcastprogramming signals may be received as a live public broadcast, or maytake the form of programming content received prior to the time of itslater public broadcast. At 105, the incoming broadcast signals areparsed or subdivided into logically separate segments, which need not becontiguous and which may be overlapping or nested. The individualsegments may be processed immediately after they are identified duringthe parsing process, or they may be stored for future processing in astorage unit 107.

[0046] As illustrated at 111, metadata is then created which describeseach of the identified programming segments. The metadata describingeach segment may take the form of a separate data entity, or may bestored or transmitted with the content of programming segment, which itdescribes. Unless the metadata is associated with a particular segmentby being stored or transmitted with that segment, it includes a pointeror some other mechanism for specifying the segment or segments itdescribes. In addition, the metadata typically includes additionaldescriptive information about the associated segment(s). The metadatacreated at 111 may be immediately processed or transmitted to the userafter it is created, or may be stored for later processing ortransmission in a storage unit illustrated at 113.

[0047] Only selected items of metadata may be transmitted to the userlocation. The specific metadata transmitted may be selected as shown at115 in a variety of ways. Data describing the demographics of individualusers and data specifying user preferences stored at 117 may be used toselectively provide the user with only that portion of the availablemetadata which is best suited to the needs of the user or which a thirdparty, such as an advertiser, desires to make available to the user.

[0048] Note that metadata created by the user, or preference datasupplied by the user or derived from an analysis of the user's use ofthe system, or from the viewer's demographic characteristics, may becombined with or used instead of metadata and preference data created atthe remote location.

[0049] Note also that the content of broadcast programming received atthe remote site may be forwarded to the user location with or separatelyfrom the corresponding metadata. This content information may take theform of the broadcast programming received at the remote site at 101,previously received programming stored at 103, and individual segmentsas parsed at 106 and stored at 107. As noted above, the metadataassociated with these programming signals may be combined with theprogramming content as transmitted to the user, or may be sentseparately over the same or a different communications pathway.

[0050] The communication methods or apparatus used to transport metadataand/or content to the user as illustrated at 130 may take many differentforms, including: the Internet, a dialup telephone connection throughthe public switched telephone network (PSTN), a wireless transmissionsystem, cable, private line facilities, or data storage mediatransported from the content publisher and/or the metadata creator tothe user. The communications may take place over a combination of suchfacilities and, as noted earlier, the content and metadata may betransmitted in one or both directions together or separately over thesame or different facilities.

[0051] Metadata created at the remote location and transmitted via thecommunications facility 130 may be stored at 133 at the user location.The metadata stored at 133 may be edited at the user location asindicated at 135, and metadata from the user location may be returnedvia the communications facility 130 to the remote location for shareduse by others.

[0052] At the user location, broadcast programming signals are receivedat 141, either in the form of a live public broadcast from the source100, or as programming content received from the remote location via thecommunications link 130. It a leading purpose of the present inventionto provide the user with a better and more convenient way to identifyand reproduce that portion of the large quantity of programming that isbroadcast for general consumption from many sources via many pathways,including conventional radio and television broadcasting, whether overthe airwaves or via a cable or satellite facility. The metadata that isprovided from the remote location via the communications pathway(s) 130may be used to selectively store, organize and/or selectively reproduceprogramming received directly at the user location from a source 100, orreceived together or separately with the metadata via the pathway 130.

[0053] The broadcast programming content received at the user locationat 141 may be immediately processed or stored for later processing andviewing. As described in U.S. Reissue Patent 36,801 issued to James D.Logan et al. by the invention, the incoming broadcast programming may beconcurrently viewed or otherwise processed while it is being recorded ina circular buffer for possible future use. A reserved portion of thestorage unit seen at 143 may implement the circular buffer. This allowsthe user to utilize VCR-type controls to pause and selectively replay orprocess previously broadcast programming at different forward andreverse playback rates. With the pause capability, the system isconstantly recording the last 5 minutes or so of a live radio broadcast,or the last 30 minutes or so of a live television broadcast. When theuser hears or views a song or program that he or she likes, the userpresses a “Catch” button, and the program will set aside the all of apredetermined part of the stored programming in the circular buffer, aswell as a further predetermined part of the incoming broadcast thatcontinues the saved portion, and retains both in temporary storage at103. Later metadata may then be applied to that segment identifying thebeginning and end of the program or song being played at the time thecatch button was activated. If the button was hit after a program orsong was over, but before or after another began, the system wouldassume the user was trying to capture the last played song.

[0054] Unless received in already parsed form from the remote location,the incoming broadcasts are parsed at 145 into segments that correspondto the segments created at the remote location at 105. As noted earlier,the available metadata may be used to subdivide the incoming broadcastsignals into segments. For example, the metadata may identify incomingsegments by source and by start and end times. Alternatively, themetadata may include “fingerprint” or “signature” signal pattern thatcan be compared with incoming broadcast signals to identify particularsegments, and may further include timing information, which specifiesthe beginning and ending of each segment relative to the location of theunique signature.

[0055] After individual segments have been identified in the incomingbroadcast stream at 145, they may be immediately processed or stored forfuture use in the storage unit 145. Not all of the segments that areidentified may be of further use; accordingly, the available metadatamay be used to select or discard particular segments as indicated at151, and to process only the remaining segments, or selectively storethem for future processing or playback at 153.

[0056] At 161, the selected segments may be modified or reorganized in avariety of ways in accordance with the metadata. For example, thesequence in which program segments are presented for playback may bemodified, and programming materials not necessarily included in with theoriginally broadcast materials may be “spliced” into the presentation,or all or part of selected segments may be deleted from thepresentation. The resulting program content which is in condition forplayback may be immediately presented to the user, or it may be storedat 163 for selective playback at a more convenient time as indicated at171 and 190.

[0057] As illustrated in FIG. 1 at 180 and 135, the user may createdescriptive metadata and may edit metadata previously received orcreated in a variety of ways to personalize the storage, reorganizationand playback of available broadcast programming.

[0058] It should be observed that the process of creating and editingmetadata may be based on any one of the various versions of the receivedcontent; that is, the content as received and stored at 143, as parsedand stored at 147, as reduced to specific segments remaining after theselection and discarding process at 151, and as modified at 161 andstored for viewing at 163.

[0059] It is also important to note that the parsing, selection andmodification processes may be performed at different times using, ineach case, the most recently stored version of the programming contentand the metadata that is available at that time. For example, metadatathat is used to parse incoming segments at 145 may be made availablefrom the parser 105 at the remote facility at an earlier time thandescriptive metadata arrives from the remote creation process 111. Thepresence of the storage unit 143 allows received broadcasting signals tobe held until parsing metadata arrives which will subdivide the receivedprogramming into logical units that can then be still later selected andmodified with the aid of descriptive metadata that arrives only after itis created by the remote editing process. Note also that the metadatawhich arrives first to subdivide the programming stream into logicalsegments, as well as available metadata which describes those segments,facilitates the task at the user location of generating still furthersupplemental metadata which describes, rates, annotates or recommendsprogramming content for other users.

[0060] In the description that follows, many of the features andfunctions summarized above and illustrated in FIG. 1 will be presentedin more detail.

[0061] Program Source 100

[0062] The present invention contemplates the creation and use ofmetadata for describing and manipulating programming content of the typetypically broadcast for public consumption by radio and televisionbroadcast stations; disseminated by cable and satellite systems and,more recently, via the Internet; or published for general consumption ondata storage media, such as DVD disks. This broadcast programming may bein analog or digital form and, in some instances, may be obtained from acontent provider prior to being broadcast. It is important to observethat the “broadcast programming” from the source 100 is available forprocessing at both a remote station and at the user's location asillustrated in FIG. 1.

[0063] The principle illustrative embodiment as described below is usedto select, organize, disseminate, store and reproduce televisionbroadcast programming. It should be understood however that theprinciples of the invention are, with few exceptions, equally applicableto radio broadcast programming, to programming that is published via theInternet, and to programming such as movies which are transported to theuser on published data storage media, such as DVD disks.

[0064] Storage Unit 103

[0065] While the parsing of programming content into segments and theassociation of descriptive metadata with those segments may be automatedto some extent as described later, it is frequently desirable to provideone or more human-operated editing workstations which can used to adjustor “fine tune” the time position of markers which delimit the beginningand end positions of segments, and to manually compose descriptivemetadata, provide qualitative rankings, and to otherwise classify ordescribe the content of each segment. The use of storage units 103and/or 107 permits unparsed and parsed programming to be temporarilystored so that it may be processed through one or more multi-channelediting stations where a single operator can effectively insure theaccuracy of the parsing process and the addition of descriptive metadatato plurality of concurrently received broadcast programming channels.These multichannel editing stations are used at 106 and 111 to subdivideprogram content into segments described by metadata. The multichannelediting stations employ variable speed playback techniques to controlthe placement of time markers, and may display close caption text in anediting window to assist the human editor in composing descriptivemetadata and classifying the content. These multichannel editingstations scan the programming content in storage units 103 and/or 105and place the resulting metadata in the store 113 for distribution tousers as discussed later.

[0066] Parsing Broadcast Programming at 105

[0067] Automated means may be used to subdivide programming intosegments. For example, segment delimiters may be created in response tothe detection of scene changes (frequently indicated by blank frames inTV content) or by abrupt changes in overall image content whenbackgrounds change. In addition, voice recognition processing may beused to detect and automatically map the times when particularindividuals are speaking. Predetermined image content may be detected toidentify repeatedly used screen displays at the beginning and end ofprograms, and at program breaks or intermissions. In the same way, audiorecognition may be used to identify standard theme songs andannouncements used at the beginning of certain program segments. Whensuch standard elements also serve as program segment identifiers, theymay be associated with standard descriptive metadata that isautomatically accessed from a library and added to form all or part ofthe descriptive metadata that is associated with the identified segment.

[0068] Frequently, when parsing television programming, the audio andvideo components have different time boundaries. For example, if it weredesired to subdivide a football game telecast on a play-by-play basis,the audio description may well begin before and extend well after theactual play as seen in the video component. If the programming wassegmented based on the video alone, the audio would be segmented in asomewhat non-optimal fashion while, in the same way, isolating a videosegment alone might cut speakers off in mid-sentence. This follows fromthe fact that commentary is not frequently not timed to occur betweenthe beginning and end of the activities shown on the video portion ofthe program.

[0069] Thus, it would be advantageous at times to split the audio at adifferent point from the video. This strategy might result, however, ininterrupting a commentary underway when the next visual segment began.As long as the audio structure does not match the video structure, thehuman editor should be provided with the ability to independently selectdifferent beginning and end points for the video and audio segments, andthen be provided with mechanisms for shortening the longer of the two,or lengthening the shorter of the two. The audio content may belengthened simply by adding one or more periods of silence to the audiostream, and may be shortened by deleting silent periods to compress thepresentation. The video presentation can be lengthened by adding “fillercontent”, by adding freeze-frame displays, or by reproducing content inslow motion.

[0070] Another strategy is to optimize the “smoothness” of the audiosplits at the expense of the video splits. Thus, instead of splittingthe video at the moment the ball is hiked, it might be split at alogical break point for the audio at some point before the ball ishiked. This might make the video a bit choppier, but audio smoother.Note that the image presentation can be more easily lengthened orshortened by slowing or speeding the display rate or by duplicating ordeleting frames, whereas the human ear would easily detects the changein pitch when an attempt is made to alter the presentation rate of anaudio signal.

[0071] Other method may be used to separate out the songs from the audiostream are by use signal analysis to distinguish music from talkover orto distinguish one song from another.

[0072] Another method used to determine a markup point which willeliminate song talkover is to estimate the likely spot of the end oftalkover by employing a database that specifies how far into a song youmust go before finding the lyrics or main theme of each song, or thepoint at the end of the song when the lyrics or main theme ends. This“start and end of music” point, would be used as a best guess as to whenthe DJ talkover stopped (or if at the end of the song, when the talkoverwas likely to start again). The DJs themselves often have thisinformation and often use it as a guide that allows them to talk rightup until the point that the music starts. For stations that continuallyemploy talkover, putting markers at these predetermined start and endpoints would provide assurance that no talkover was played with thesong.

[0073] It should be noted that segments are not necessarily thecontiguous results of subdividing the original programming signal.Segments can be unique, or can overlap or be nested within othersegments. Moreover, a segment is not necessarily a subpart of anindividual program as broadcast. A segment may be a combined collectionor sequence of such programs, may correspond precisely to a singleprogram as broadcast, or may be only one part of a longer program.Importantly, segments may be organized into groups of other segments orprograms, and can form a hierarchy of sections, chapters, sub-chapters,etc. Thus, a single metadata entity may be associated in a variety ofways with a plurality of segments, while other metadata entities may beassociated with only one segment.

[0074] Storing Parsed Segments

[0075] The programming content that is stored in discrete segments at107 need not be a direct reproduction of the incoming program signal.Redundancies and overlapping content may be advantageously removed. Forexample, when audio content is stored, the periods of silence may beremoved for more compact storage. The content signal may also becompressed if desired using, for example, MPEG compression for video andMP3 compression for radio broadcast programming. A linear programmingprocess may be used to allocate segments for the scarce viewing timeallocated by the user.

[0076] After the continuous broadcast data has be assembled intoindividual segments, it is frequently preferable to store those segmentsso that the descriptive metadata which describes each segment can becreated, automatically or by a human editor, or both, and be associatedwith the program content at a pace which is independent of rate at whichthe segments were originally broadcast. The nature of the descriptivemetadata as thus created is described next.

[0077] Creating Metadata at 111

[0078] First, it should be noted that if the metadata is notpositionally associated with the segments it describes by being imbeddedwith, or transmitted at the same time as, the content data, some of themetadata performs the function of identifying the associated programcontent.

[0079] Stored segments may be identified by a file name, a URL, or bysome other unique access key (such as the primary key value in arelational database storage system). When segments can be identified andaccessed when needed using such an access key, simply including that keyvalue with the descriptive metadata suffices.

[0080] However, when metadata created at the remote location must beassociated with program content received at the user location, adifferent mechanism is needed. As one approach, the program segment maybe specified by the combination of an identifier which specifies abroadcast program source (e.g. a particular broadcasting station orcable channel) together with the start and ending times at which theparticular programming segment was broadcast. These “time stamp” valuesare sent with the metadata to the user location and matched against timestamp information associated with the broadcast programming whenreceived at the user station. For example, a TV program segment may beidentified by data indicating the segment was broadcast by WGN beginningat 11:23:42 to 11:32:16 GMT on Oct. 12, 2000.

[0081] At times, predetermined time shifts occur when programs aredistributed over cable facilities and the like. When that occurs,predetermined time offsets can be added to or subtracted from the valuesspecified in the metadata, either before or after the metadata istransmitted to the user location. The magnitude of these standardoffsets may be determined by detecting the time when predeterminedsignal patterns are received at the user location, comparing that timewith the time when that signal pattern was broadcast as measured at theremote station to generate the offset value to be applied to allsegments experiencing the same time shift as the predetermined signalpattern.

[0082] The technique of detecting predetermined signal patterns may beused to establish not only the timing but also the identity of a segmentof a sequence of segments. For example, one or more a unique“signatures” may be extracted or derived from a sequence of programmingsegments from a particular source. The metadata for individual segmentsmay then include values that specify a time offset from the signaturemarker and, in that way, uniquely identify the segment.

[0083] The technique of identifying segments by means of “signatures”may be used when the stream from which the metadata was derived differsfrom the stream recorded by the consumer. For instance, if a localbroadcast changed the timing of a broadcast program in order tointroduce advertising of different lengths, or to add locally focusedcontent not included with the version from which the metadata is made,problems would arise. As another example, the metadata might describesegments within a pay-per-view movie that might be received at differenttimes by different users. In this case, “signature” or “content-basedtime stamps” may be used to associate metadata with the stored contentunder these circumstances.

[0084] When the metadata is created, a “signal pattern,” or“fingerprint” extracted or derived from the content is used to identifya known time position in the “parent” copy of the version from which themetadata is created. This fingerprint or pattern may also uniquelyidentify the parent copy, distinguishing it from other content. Thisfingerprint exists at a measurable time offset from an “index point” inthe parent copy used to associate metadata with the content. Forinstance, if the metadata were marking the beginning of an advertisingsegment, the fingerprint should be within and near the start of thatadvertising segment. Alternatively, the fingerprint to be detected toestablish the time mark may be within only the first of a sequence ofsegments, with the first and remaining segments having start and endtimes expressed by offsets from the single time mark.

[0085] Metadata used to subdivide programming content may take a varietyof forms. It may specify the position of markers, which delimitindividual segments within a programming sequence by, for example,specifying byte offsets in a file of digital programming data, or byspecifying the time position relative to some reference time whensegments begin or end.

[0086] Alternatively, metadata may specify identifiable signalcharacteristics or “signatures” within a programming signal stream.These signatures may be detected to establish the time or data positionof markers that may then be used as a base reference for data or timeoffsets which delimit the programming into segments. Such identifiablesignal characteristics may occur naturally within the programming (suchas scene changes in a video signal indicated by blank frames, theappearance of a new voice or other detectable signal pattern, or periodsof silence) or may be created by ancillary signals inserted into theprogram stream or in a parallel transmission to serve as markers. Suchancillary signaling may take the form of as identification tones,framing signals, digitally expressed data, and the like.

[0087] Using pattern-matching techniques, each piece of content storedat 103 or 143 may be compared to a specific fingerprint signature. Whena match is found, segments occurring before or after the matchingpattern may be identified at both the remote site where metadata iscreated and at the user location where metadata from the remote site isassociated with the corresponding received broadcast segment. Multiplefingerprints may be used in order to continually synchronize the twoversions.

[0088] The viewing habits of users as revealed by usage logs may beanalyzed to subdivide programming into logical segments. With a largeenough base of users, a profile of viewing could be constructed for agiven program which would tend to indicate when users skipped particularsegments, or used the mute control to silence a particular segments, andto further identify segments which held viewer's attention. This type ofobserved behavior could be combined with other techniques, such as blankframe, scene and voice change detection, and analysis of the closedcaption text to further automatically determine the boundaries betweenlogical segments.

[0089] The segment boundaries chosen by automated techniques may berefined by a human editor who makes adjustments to the timing of theautomatically selected boundaries as needed. Thus using automatedtechniques, it is possible to subdivide broadcast programming intological segments and to provide a figure of merit rating which can besent to those who view the same programming on a delayed basis to assistthem in making program viewing and recording selections, and, ifdesired, to automate those selections in whole or in part.

[0090] Storing Metadata at 113 and 133

[0091] As noted earlier, metadata describing the segments identifiedduring the parsing process at 105 may be created at 111 in a variety ofways and stored at 113 for potential distribution to users. In addition,metadata created by users may be received via the communicationsfacility 130 to supplement or replace the metadata created at 111.

[0092] Metadata created by users may be shared directly between users.When shareable metadata exists at a user location, it may be“registered” by supplying its resource address (such as an Internet URL)to the remote location which then relays the URL to other users whodirectly access the descriptive metadata from the other user's metadatastorage 133 in a peer-to-peer transfer. In this form, the remotefacility shown in FIG. 1 operates as a registry or directory thatpermits users to share descriptive metadata about broadcast programmingwith one another on a community basis.

[0093] The remote facility may subdivide available broadcast programminginto segments as previously described and then associate each segmentwith references or pointers to metadata created by users and hosted onuser's computers or on an available storage resource (including, forexample, storage space made available at 113 for storing metadata).

[0094] As an alternative, the metadata provided by users may includesegment identification information. For example, a user may identify asegment of programming by marking its beginning and end, and then createmetadata, which describes, rates or classifies that segment. Programmingat the user location creates identification metadata for the segmentusing any of the techniques discussed earlier; for example, byextracting and transmitting a unique fingerprint from the identifiedprogramming and transmitting this fingerprint together with start andend offsets, or by identifying the programming source together with thetime stamp information specifying the times at which the beginning andend of the segment were originally broadcast.

[0095] The user may review metadata supplied by other users andpresented as a program guide to the available stored programming. Beforethe descriptive metadata from other users is displayed, the segmentidentification portion of the received metadata may be compared with theprogramming content stored at 143, 147, 153 or 163 (or with metadatastored at 133 which identifies the content available to the user). Inthis way, only that descriptive metadata from other users whichdescribes available programming need be reviewed.

[0096] Alternatively, a viewer may transmit a request to the remotefacility for additional information about a particular program (whichmay include multiple segments), or the preferences of the user as storedin 117 may be expressly stated by the user or derived from the user'sviewing history. These requests and/or preferences stored at 117 maythen be used at 115 to select desired metadata (including references tometadata stored elsewhere) in the store 113 for transmission to therequesting user.

[0097] Thus, the metadata which is created by created by and sharedamong users one or a combination of the following forms:

[0098] 1. Qualitative (rankings, reviews, etc.);

[0099] 2. Descriptive (summary, topics, etc.);

[0100] 3. Segment identifications (start time, elapsed time, endingtime, source, detectable characteristic, ancillary codes); and

[0101] 4. Cross-references or pointers to metadata stored at addressableresource locations, including metadata created and hosted by otherusers.

[0102] Metadata that includes the URL of a World Wide Web resourceprovides a robust mechanism for associating the content of particularsegments of broadcast programming to both additional information andrelated interactive transactions. For example, metadata may beassociated with programming that permits viewers to learn more about orto purchase products or services related to the programming content. Asdescribed above, individual users may also create addressable resources,such as Web pages, and associate links to those resources with viewedprogramming segments. For example, a fan club for a particular actormight create a Web site devoted to that actor, and then share metadatacontaining the URL to that Web site with other viewers.

[0103] The user's ability to create and share metadata that describes,classifies or relates to selected broadcast programming segments thusenables users to create a community surrounding those segments in whicha rich variety of information exchanges and transactions can occur.Users can, in effect, use the subject matter of broadcast programming aspublic bulletin board upon which to post comments about the program,ratings and descriptive data which can be used as a basis for indexingand retrieving program content, and for linking in related informationfrom other sources, or for conducting a marketplace by posting offers tosell and to buy goods or services relating to or suggested by programcontent.

[0104] A “Community Markup” system (here called “CM”) may be implementedthat serves two purposes: it may be used as a way to develop markup datafor sources of program information that may have an insufficiently largeaudience to justify the creation of markup by a commercial enterprise,or to improve the quantity or quality of markup data offered bycommercial sources.

[0105] To optimize the benefit of the community markup, program guidedata may be made available to potential users to identify what stationsto record. As users can't go back after a broadcast and record it, thismethod would insure the maximum number of recorded copies will beavailable both for markup and playback with any CM effort.

[0106] CM can also be used to improve previously produced markupinformation. For example, if the markup does not accurately reflect theextent to which an announcer may have “talked over” a song, users willhave editing tools available to them to alter the placement of the songdelimiters and excise the talkover. The CM system will allow users tojoin a community whereby they will be able to upload their improvedmarkups to a central server at 113 so that other users may access them.A “barter” system may be employed so that, when a user creates originalmarkup data, he or she would then be entitled to receive markup datafrom other users, potentially avoiding the free-rider problem.

[0107] Improved markups may be downloaded and used to improve previouslyrecorded songs or other content stored at 143, 147 or 153 in anautomatic mode. Thus, even if several days elapse before the improvedmarkup is available, the existing recording library would beautomatically upgraded. This upgrading of the library would be performedtransparently to the user.

[0108] As the originally recorded material is still in local storage,and only the metadata defining the playback markers is altered as aresult of the new metadata, the recipient of community improved markupcould always “undo” the automatic marker movement and restore theoriginal recording and associated splits.

[0109] In cases where the system receives multiple markups, they can beaveraged together for greater accuracy with outliers being deleted. Thisaveraging function can be performed either at the server based onmetadata received and stored at 113, or at the user location based onmetadata received and stored at 133.

[0110] Note that community created markup would not necessarily have tobe stored on a central server. Markup data can be stored solely on usermachines and shared via peer-to-peer transfers (e.g. using anarchitecture of the type employed by Gnutella). In this environment,users would employ shared directory, which would identify metadata abouta recording they had made, which exists in storage at 133 at anotheruser location.

[0111] Community Markup (CM) may be created as a byproduct of the user'suse of locally generated metadata for creating a personalized programlibrary. For example, the user may record a lengthy radio broadcast froma favorite station and then selects particular songs for inclusion in apersonal library, either by using markup signals provided by an remotemarkup source, or by using the available editing tools at 135, the songswhich are identified may contain DJ talkover at the beginning or end ofthe song. In that case, the user may be employ a one-step-editingfeature that permits the user to listen to a song and, when a transitionoccurs from talk to music, or visa versa, they can simply click on a“scissors” button which moves the start-point or end point of play, forthat song, so that, the next time it's played, the new start and/or endpoint takes effect. Importantly, the talkover is not erased and the playmarker is merely moved. If the user did not time very well the use ofthe scissors, he can hit an “undo” button and redo the clipping process.

[0112] At any time, the user may elect to share the locally storedmarkup signals with others by transferring that markup to the server forstorage at 113 where it is combined with the markup produced by othersfor that station and time frame, or by transferring the markup toanother user with a peer-to-peer transfer. In this way, not only is themarkup shared which accurately identifies desirable programming, themarkup also operates as a recommendation for that section and, whenaggregated among many users, offers the ability to identify and sharethe “best of today's programming” on a particular station.

[0113] A mechanism related to the “scissors” function described abovewould enable a given program segment (e.g. song) to be “split out” fromthe original program recording. Because the available metadata may notaccurately identify the precise beginning and end point of a given song,a predetermined duration of programming is included at both thebeginning and end of each song as identified by the preliminarymetadata. This extra time provides “running room” to make sure thatevery program has at least the entire rendition in it. Since this extralength could include material from the program segment behind or aheadof the segment being edited, the interstitial material in the nebulousspace between songs is duplicated and added to both songs as defined bythe metadata. The user may then user the editing means, including the“scissors” function noted above, to provide a final adjustment to thestart and end time. When program segments are permanently stored in aselection library, the added material excluded by the final edit maynonetheless be retained at both the beginning and ending to preserve theability to adjust the start and end points even after the selectedprogram segment is persistently stored in the library.

[0114] Programming may be described, classified and rated using metadataformats. Standard rating systems have been widely promulgated using theWorld Wide Web Consortium (W3C) Platform for Internet Content Selection(PICSJ). The PICS specification enables labels (metadata) to beassociated with content and was originally designed to help parents andteachers control what children access on the Internet, but alsofacilitates other uses for labels, including code signing and privacy.PICS labels, and other metadata, may be advantageously expressed usingthe W3C's Resource Description Framework (RDF) which integrates avariety of web-based metadata activities including sitemaps, contentratings, stream channel definitions, search engine data collection (webcrawling), digital library collections, and distributed authoring, usingXML as an interchange syntax. Details tutorial information and formalspecifications for PICS and RDF are available on the World Wide Web athttp://www.w3.org/pics/ and http://www.w3.org/RDF/respectively.

[0115] Storing User Data at 117

[0116] Whether the metadata which relates to programming segments iscreated at the remote source or at one or more user locations, it isfrequently desirable to organize or filter the metadata so that theyuser can more easily obtain the benefit of that metadata which best fitsthe needs or desires of the individual user.

[0117] One mechanism for limiting the amount of metadata actuallypresented to the user is to simply store all received metadata at 133and then to employ means for sorting and/or indexing the stored metadataso that desired metadata can be located in response to the user'sspecifications. As an alternative, the user's specifications may beuploaded via the communications facility 130 and stored at 117 at theremote facility. The user's specifications or preferences as stored at117 are then used at 115 to select only that metadata which best fitsthe user's needs for transmission to the user's metadata storage at 133.

[0118] The user's preferences may be derived from his or her activity.For example, the particular programs a user chooses to save or view maybe monitored to determine the user apparent content preferences.Preference data may be produced at the user's location and stored withother metadata in the store 133, from which it may be used locally orsent to the remote location for use there. Alternatively, “user log”data recording the user's activity may be transmitted to the remotelocation where it is analyzed to produce preference data.

[0119] Metadata which is derived from an analysis of the recordedviewing or editing choices made by other viewers, which may be termed“implicit metadata,” includes values such as: the number of users withwhom the viewer had common tastes who watched a particular program, ormetadata based on analyzing such events as (a) who surfed out of, or didnot complete watching, a certain show, or never recorded it in the firstplace; (b) who took a certain amount of time to watch the recording (ifit's a preferred program to a viewer, it will be viewed sooner; or (c)what percent of the program, on average, was skipped.

[0120] Once the preference data are determined, they may used in avariety of ways:

[0121] a). Preference data may be used at 151 to select or discardparticular received broadcast segments so that only those which are morelikely to be of interest to the user are saved, thus conserving storagespace;

[0122] b). Preference data may be used at 161 to modify program contentby, for example, inserting, interleaving or substituting advertising orother materials with received program materials based on the usersinterests;

[0123] c). Preference data may be used at 171 to assist the user indetermining which received segments to play, either by automaticallypresenting those segments most likely to be of interest to the user, orby presenting a program guide containing or highlighting segments ofinterest from which the user makes the final program selection;

[0124] d). Preference data may be used to help the user select programsegments which are made the subject of additional, user-created metadatawhich is then used locally (e.g. bookmarks or notes for the user's ownuse) or uploaded to the remote location or shared with other users asnoted above;

[0125] e). Preference data may be used at the remote location where itis stored at 117 and used at 115 to select metadata for transmission tothe user;

[0126] f). Preference data for individual users or combined preferencedata from many users may be used at 103, 107 and 113 to determine whichprogramming content and descriptive metadata should be stored, and whenpreviously stored content and metadata should be discarded, to make themost efficient use of limited storage space in light of user demand; and

[0127] g). Preference data may be collected based on the usage of, orratings supplied for, the metadata itself. In this way, users may ratethe perceived value of metadata created automatically or by the editorsat the remote facility (at 111) and this rating data may then be used at115 to select not only programming of particular interest but also toselect the metadata deemed to be of the most value.

[0128] Note that the metadata created at 111 and/or 180, and stored at113 and/or 133, may include metadata used to display an electronicprogram guide (EPG) for the user which displays in some convenientformat information concerning the content of available broadcastprogramming. Such displayed metadata associates items of descriptiveinformation with one or more program segments. It is thus frequentlyadvantageous to provide the user with means for associating user-createdcomments, notes, reviews, ratings, and the like by using the EPG displayto identify and select the program segments with which the newly createdmetadata is associated. Metadata created in this way is thus readilyshared with other users who share comparable EPG metadata by the simplemechanism of permitting a user to request additional information about adisplayed program guide item.

[0129] As noted earlier, metadata created by individual users may besimply stored locally at 133 as an Internet accessible resource. Webcrawling “spider” programs executing on remote computers may thenretrieve and index this metadata and then act as “search engine”directories that may be publicly accessed to locate metadata ofinterest. For example, a search for “Stardust” might locate metadatadescribing an audio recording of the song by that name, biographicprogramming about the composer or performing artists, and the like.Thus, the descriptive metadata created by professional editors and/orusers can form the basis for finding and enjoying content that wouldotherwise be difficult to index because of its non-textual character.

[0130] Metadata can be developed to characterize individual programsegments by processing log file data representing choices made by usersin selecting and/or abandoning programs, and from program ratingsexpressly provided by users. When aggregated by retrieving and combiningsuch data from many users, and when further correlated with demographicdata about the same users, rating information can be provided whichtends to indicate what other viewers having similar backgrounds andsimilar past preferences preferred among the currently available programmaterials.

[0131] Ratings data compiled from actual user selections may provideunique information on how specific consumers react to specific songs atspecific times. Thus, a recording studio might release a new single, andimmediately thereafter determine how many listeners in a certaindemographic had deleted, saved, or listened to that song multiple times.Express song rating data provided by users could be used in addition toor instead of implicit rating data to identify specific program segmentsthat were well received or uniformly disliked.

[0132] When programming is broadcast in one geographic area before beingbroadcast in another, or when programming is repeated, the viewing andlistening behavior of users exposed to the earlier broadcast can be usedto provide rating information for later users. Thus, the habits of TVviewers on the east coast of the United States could be analyzed inadvance of the later rebroadcast of the same programming on the westcoast, so that ratings data tending to reflect which of the programswere preferred may be supplied to west coast viewers in advance. Inaddition, west coast viewers would have the benefit of advance reviewsand summaries of programs created during the earlier broadcast. In thesame way, any viewer using a personal video recorder (PVR) or othermeans for accessing program materials on a delayed basis could be aidedin the selection of that program which they, as individuals, would bemost likely to enjoy by the availability of rating and review metadatafrom earlier viewers having similar interests.

[0133] Content and Metadata Communications

[0134] The transfer of both content and metadata is illustrated at 130in FIG. 1. As described here, both the remote location and the userlocation may receive and process programming signals (content) from abroadcast programming source. In addition, content may be sent from theremote location to the user location, and content may also be sent fromthe user location to the remote facility or to other users on apeer-to-peer basis. By whatever path is used, the content which ispresented to users is made available to a large number of potentialusers, and the metadata which describes that programming material iscreated to aid those users (or particular users) to selectively recordand view this programming material.

[0135] The metadata may be created at the remote facility andtransferred on a selective basis to individual users, or it may becreated by users and transferred to the remote facility forredistribution to other viewers, or it may be transmitted directly fromuser to user on a peer-to-peer basis.

[0136] The metadata may be transmitted with the programming content, ormay be transmitted at a later time, or over a different communicationpathway. In many program transmission systems, some of the availablebandwidth is allocated to metadata, as typified by program guidechannels or time slots provided by the vertical blanking interval (VBI)in a television signal. These existing pathways may be used to transferthe metadata contemplated by the present invention which contemplates,in many implementations, the transfer of metadata after the programmingmaterial has been broadcast but before the programming material isviewed on a delayed or time-shifted basis after having been recordedearlier.

[0137] Thus, as described here, the metadata may be created by editorsor viewers who comment on or rate viewed material at the time of orafter its initial broadcast, with the metadata being transferred to endusers to facilitate the selection, recording and playback of desiredmaterial on a time shifted basis. In summary, the metadata flow need notbe transmitted before or concurrently with the original broadcast, butis may be created by early viewers and used by later viewers who watchthe programming on a delayed basis, either because the version theywatch was broadcast later or because the version they watch waspreviously recorded for later viewing.

[0138] Creating and Editing Metadata at the User Location

[0139] As previously noted, metadata may be both created (at 180) andedited (at 135) at the user location. The user may programmaticallyderive this locally created metadata from the viewing choices madewithout requiring any additional effort by the user, or the locallycreated metadata may be the result of interactive choices made by theviewer. For example, a viewer may receive metadata from the remotesource which takes the form of an electronic program guide describingbroadcast programming, and with respect to each item of suchprogramming, the locally generated metadata may indicate whether or notgiven program segments had been (a) selected for storage for potentiallater showing, (b) selected for actual viewing, (c) viewed for aspecified period before being terminated, (d) saved for later repeatviewing after having been viewed, (e) expressly rated by the viewer, or(f) made the subject of a written text review. This locally generatedmetadata reflecting the user's use of or assessment of the programmingmaterials, as placed in storage unit 133, is then uploaded to a remoteprocessing site for distribution to other viewers or simply placed in anaddressable location from which it may be retrieved for processing byone or more rating services or by other viewers.

[0140] When the user stores broadcast programming in the store 143, theuser has no control over the incoming content. To more easily controlwhat is saved for possible future playback, the user may be providedwith a “Never Again” button. Whenever the user is listening to orediting a program segment, such a song, or has highlighted that programin a library program listing, the user may press the Never Again buttonto prevent that song from being recorded or, if recorded, toautomatically prevent that song from being presented to the user in alist of available songs. Alternatively, pressing the Never Again buttonmay also permit the user to prohibit the listing of any song by aparticular artist, of a particular song by any artist, or furthereditions of a serialized program.

[0141] Over time, the use of the Never Again button may be used todevelop a “negative screen” of preferences for that user and may be usedto automatically eliminate or reduce the number of program segments orsongs related to a program song excluded by the Never Again button. TheNever Again button may also be one of the several ways that users willbe able to accumulate preference information that can be used to controlplaylists transmitted from the server or created locally by the user.Note that, like other metadata, the Never Again list is kept as aseparate file and users may undo a Never Again designation at any timeso that it will have no further effect on existing or future recordedcontent.

[0142] Instead of a negative filter, a huntlist, or “positive” filtermay be used as well. With a huntlist, a user identifies which songs orwhich artists he wants the system to capture. In addition, a huntlistmay contain “songbots” (algorithms that search for described types ofsongs that the user wishes to have captured). A typical songbot could be“All Top 40's from the “70's”. Other huntlists may be created usingcollaborative filtering techniques. Huntlists may be compared withmetadata developed at a remote server (with the comparison occurring ateither the server or the user location) to flag desired songs as theyarrive from the broadcast source and are stored at 143, or they may beused a sieve, whereby hunted songs are saved and non-hunted are deletedand never presented to the user. When songs are “found” by a huntlistcreated at a remote server, an email may be generated telling the userthat new songs are now in the jukebox. When the huntlist operateslocally, a dialog box or the like may be used to alert the user to thepresence of the desired song or program segment. In addition, the usermay access his huntlist and see through a visual cueing of some kind(different colors for instance) which songs have been captured and whichhave not.

[0143] The huntlist may be compared with metadata describing theprogramming broadcast by a plurality of different stations to identifystations and times when desired programming is most likely to occur, anda program controlled tuner may then be used to automatically capturebroadcast content from the identified stations at the identified times.When program segments or songs identified on a huntlist are availablefor purchase, a “Buy” button or a “Sample” button, which allows a userto hear a sample of a song, may be presented to the user to enable thepurchase to be evaluated and executed if desired.

[0144] Automatically Bookmarking Programming Content

[0145] The system contemplated by the present invention may furtherinclude a mechanism at 180 for automatically defining and generatingbookmarks which may be applied to the content stored at 147, 153, and/or163 to facilitating navigation of the stored content and/or forpersonalizing content as performed at 151, 161 and/or 171 to therebyselectively control the playback of programming materials at 190.

[0146] The leading objectives of the automatic bookmarking mechanismcontemplated by this aspect of the present invention are to:

[0147] 1. Automatically specify segment start and stop delimiterpositions (at 145);

[0148] 2. Automatically categorize the segments;

[0149] 3. Automatically create descriptors for the segments;

[0150] 4. Automatically eliminate redundancies if necessary at 151and/or 171; and

[0151] 5. Automatically concatenate related pieces of a story at 151,161 and/or 171 to implement one or more different ways to watchtelevision

[0152] Content that is well suited for these bookmarking techniquesconsists of segment-able programming like news, sports, or shoppingprogramming, but some techniques apply to other types of programming aswell. The automatic bookmarking mechanism may be implemented with avariety of available technologies, including natural languageprocessing, voice recognition, face recognition, sound recognition, andprobability theory.

[0153] The bookmarking system can operate on the client side as notedabove, or at 111 at the central facility (which can include at thebroadcaster's facility) to create bookmarking metadata that maythereafter be downloaded to the client with the program (if the analysiswork is done ahead of time) or via a separate channel such as theInternet. The bookmarking metadata may be created ahead of time beforebroadcast, or more likely, after the broadcast when there is a shortwindow of time to create metadata before the viewer watches time-shiftedmaterial.

[0154] Creating Bookmarks From Close-Captioned Text

[0155] The preferred system may make extensive use of theclosed-captioned text. The close caption text will be feed into aNatural Language Processing Engine (NLPE) in order to interpret themeaning of the material. When the system determines a change in topic, amarker is set. The system will also attempt to categorize the materialand generate a short “slug” describing the material.

[0156] The closed caption material is typically fed into the NLPE systemin blocks, as the system can process the material faster than it isbroadcast. As a topic break might lie close to a break between blocks,the system processes overlapping blocks as needed to be sure no breakscame between, or close to the endpoint of a block. The close captiontext, when fed through the NLPE, may also be used to generate a captionfor each individual segment as well as to categorize the segment.

[0157] Closed captioning can be done live or ahead of time. When doneahead of time, it is synchronized quite tightly, within a fraction of asecond, with the program content. For live captioning, tightsynchronization is not typical, and the delay can be on the order of afew seconds. When loosely synchronized caption exists, the system mayautomatically attempt to re-synch the captioning with the video afterrecording. One way to do this would be just to use some measure ofaverage delay for that type of content and adjust the captioningaccordingly. A better method employs face recognition or shaperecognition to analyze the video content to determine when a person isspeaking by focusing on lip movements. The captioning could be re-timedto match up with the end of a speaker's as often as needed. Alternative,voice recognition could also be used when the captioning reflects thespoken sound track. Note that the accuracy of the voice recognitionwould not have to be very high since, if a definitive match was foundevery few words, the time delay could be re-adjusted until a subsequentmatch is found.

[0158] The bookmarking mechanism may use speech recognition incombination with a database of navigational words that commonly indicatethat a break or segue is in process. These would include words orphrases such as “coming up next”, “next week”, “Over to you, Bill”, etc.“When we return” would signal the start of an ad. Questions might oftenindicate a change in direction of the content. When such a phrase waslocated, a marker would be generated. Alternatively, the closed captiontext may be scanned, or using voice recognition software may be used toprocess recorded speech, to find these words and phrases.

[0159] The manner in which users view a given program may be monitoredto position automatically generated bookmarks. The video playback systemtypically includes a fast-forward mechanism that permits a user torapidly search through a program until a passage of particular intereststarts, at which time the user returns the player to normal viewingspeed. Typically, the image can be seen during this movement andsometimes the audio can be heard as well, particularly if it istime-scaled to give the audio pitch control. This fast forwardingactivity may be monitored to identify the beginning point of a segmentof interest. The system is preferably able to collect and aggregate suchbookmark position data on an anonymous basis, perhaps just from aminority of the total users, to identify the points in each piece ofcontent where users frequently resume normal playback speed afterfast-forwarding to a desired position. Note that, in general, theimportant bookmark to get right is the beginning of a segment. The endof a segment normally takes care of itself as people often skip outbefore getting to the end, or if not, the end of one segment becomes thebeginning of another. The point is that few viewers fast forward to enda given passage, but rather fast forward through a segment or sequenceof segments until the beginning of the next desired segment is reached.Due to this fact, time scaling is a useful tool for finding segmentbeginnings. This is because some number of users will scale forwardrapidly, still understanding most of what is being said if the audio isable to be heard, or will be able to view a fast motion version of videoprogramming, and will then slow down when the interesting materialstarts to play. It is this inflection point we are looking for. It willindicate a change in interest level to most viewers and thus could serveas a source of auto generated bookmarks. The preferred systemaccordingly aggregates time scaling commands from a large number ofdifferent viewers to deduce segment beginnings as an average of aconcentrated group of these fast-to-normal transition times.

[0160] Viewers will typically overshoot the actual beginning of the nextsegment as they cannot discern that a new segment has started until theywatch or listen to a bit of it. Some percentage of viewers may go backand try to start at the exact beginning, at least for some segments. Asa result, the best way to fine-tune the estimate of the location of thesegment beginning point would be to estimate the average overshootingerror and subtract that distance from the deduced segment based on theaverage or calculated segment beginning. This average-error-length couldbe found through empirical study, or deduced, by again, monitoringviewer behavior on the system and watch that small number of viewers whogo back and rewind to get to the exact beginning of a segment. Ingeneral, the system described by this invention would wish to err on theside of starting the segment too early as opposed to starting within thesegment.

[0161] Since large numbers of people fast-forward through ads at highspeeds, aggregating the data around these clusters (dropping out userswho are obviously fast-forwarding past the ad itself) would give a goodindication of where the ad stopped. Since the average user stops acertain number of seconds after the ad ends, this average stop time,minus the average error, could be used to deduce the end of the ad andstart of the next segment.

[0162] By the same token, the aggregation of data which distinguishesprogram segments which are frequently skipped by fast forwarding fromthose that are viewed normally can be used to identify popular segments.For example, a substantial number of viewers may fast forward throughthe Tonight Show to find the Top Ten segment. The system can learn tospot these clusters of segments or content through which other viewershave fast-forwarded, label them and pass them on to other users allowingthem to skip over these unwanted segments instead of fast-forwardingthrough them.

[0163] In particular, Hot Spots would be most interesting if thecomparative group was matched the profile and preferences of the viewer.This would give it more of a collaborative filtering capability.

[0164] Another form of metadata that could be automatically generatedfrom other viewers' actions is which segments elicited an interaction byother viewers of iTV functions. These might include an interaction witha Wink-like system whereby sport statistics are available over the datachannel of the cable operator. For instance, a viewer might wish tofocus on segments in a History Channel program where other viewers hadaccessed background information. Another example involves t-commerce andsystems that allow viewers to purchases an item from the TV using theremote. In a home shopping channel, this sort of metadata could serve toguide a user to the hottest items to buy.

[0165] Sound Cues

[0166] For purposes of this section, it is useful to define a new typeof content here called “rolling content.” Whereas segmentable contentincludes news and weather and linear content includes shows like“Friends” and movies like “Gladiator,” rolling content would includeprogramming such as a soccer or hockey game, which is a hybrid.Programming with rolling content have periods of higher and lowerinterest, and some climaxes like goals, but the “breaks” are more analogin nature. Many cues indicative of a break in rolling content could bededuced by sophisticated audio recognition. Important sound recognitiontypes would include laughter, applause, referee whistles, and crowdnoise. Crowd noise for instance increases dramatically every time a homerun is hit, or a shot on goal is taken in a soccer game. The systemcould understand how long it takes on average to develop a play insoccer that would cause a cheer and drop a marker in several secondsbefore each instance. In a comedy program like the David Letterman show,the “action” is expected to be continuous, so a marker would be droppedin after each instance of laughter, presuming that is the beginning ofthe next joke. A software algorithm might detect other types of soundinformation such as the level of excitement in a speaker's voice, or thequickness of speech. These variations could be transformed intobookmarks. Different algorithms could be developed for each type ofsound, and vary by each show. The user could do further modification ofthe algorithms, for instance, deciding to watch 10 seconds rather than30 seconds of content leading up to a crowd's roar. Alternatively, thesystem might “learn” preferences such as this by monitoring the specificuser's use of the fast forward button or time scaling feature.

[0167] Recognizing Repeating Patterns

[0168] Multiple copies of the same show may be analyzed to see ifpatterns repeat themselves. These patterns might be in the video oraudio and might signal the beginning of a segment. For instance, theappearance of the weather map might indicate the beginning of theweather report. The system would look for pattern markers that werespaced apart about the length of an expected segment. As stated earlier,the time scaling or fast-forward usage information could be used toconfirm that these bookmarks are usable. In addition, if nobody isskipping forward to them, that tends to indicate they might not becorrect.

[0169] Music Recognition

[0170] A music discriminator, that is a signal analyzer able todiscriminate between music and talk and deduce when music is playing inthe background, can be used to provide bookmarks. Music analysis mayalso be used to distinguish one type of music from another, and perhapsdistinguish bands or songs. These algorithms could be useful fordetecting breaks in a video show, as well. This technique could beparticularly useful for detecting ads as many start with music.

[0171] For rolling content or linear content, detecting the type ofmusic playing might be useful. In many cases, music is used to highlightthe “essence” of a movie. In many movies, a characteristic type of musicplayed during each action scene or love scene, for instance. Metadatabased on the type and location of this background music could be used toclassify areas of content into different moods or types of content suchas love scenes, action scenes, etc. A user could use this information tojust play back these portions of the content.

[0172] Another form of similar metadata would be the frequency of scenechanges. More scene changes would indicate more action. By the sametoken, the degree of motion in the image itself can be detected from theamount of redundant information that is dropped out in the encodingprocess. This information could be used to deduce or measure the degreeof motion in the scene, information which could be used to deduce the“action level” in the scene, perhaps in conjunction with otherindicators.

[0173] Character Changes

[0174] The preferred system would be able to detect the coming and goingof characters, announcers, or actors in the programming. This could bedone through face recognition technology or through voice recognition(where different peoples' voices are recognized regardless of what theyare saying). In news shows, this would be particularly useful when oneannouncer hands off to another. For other types of programming it mighthelp to automate our “Favorites” Way to Watch (which we typicallydescribe as a way to track Tiger Woods through a golf tournament).Further logic, implemented using data generated from voice or facerecognition, may be used to determine who was the anchor and who werethe subsidiary reporters. The breakpoints could be focused on the timeswhere the camera went back to the main announcer.

[0175] Visual Cues

[0176] Scene recognition (as opposed to scene change recognition) wouldbe useful in deducing breakpoints. Similar to sound recognition, visualrecognition could (either now or in the future) spot when twonewscasters were using a split screen, when stock prices were on thescreen, when a ball went into the basket, etc. Overall visual metrics,such as the amount of movement on a soccer field, could be indicative ofa timeout or frantic action.

[0177] User-Generated Bookmarks

[0178] Another form of non-tagging-station-generated bookmarks would befor users to create their own bookmarks by clicking a button as theywatched the programming. This could be related to the Save feature (seebelow), or merely to enjoy while re-watching the programming. The usercould also have the option to categorize and label the segment if bothbeginning and end points were denoted. If enough users in the monitoredsampled bookmarked the same scene, the system could average theselocations out to present a definitive mark to other users. In the sameway, when a user bookmarks a spot in order to save a segment, this newviewer-generated location data could be used to create the deducedbookmark.

[0179] There are four types of viewer-generated bookmarking information:Fast-forwarding or other analog motions through the video, bookmarkingfor later repeat viewing or showing to others, bookmarks made to send toa friend, and bookmarks made for purposes of saving. In this list,viewers can be assumed to have the most thought into the actions laterin the list. Viewers saving segments would therefore be presumed to haveput the most thought into the exact placement of the bookmark. As aresult, as data from these multiple sources is compiled and synthesized,extra weight would be put into the latter categories. The exactweightings could be tested through empirical testing. That is, an editorcould study the video and determine the “proper” bookmark locations andthen develop a model for using these data inputs in the most accuratefashion.

[0180] Note that once “deduced bookmarks” start to be presented toviewers, the system would cease to collect as much new information asviewers' navigation actions would then be based on the data beingsupplied. Therefore, the system would have to decide at what pointenough field data had been collected before dispersing its deducedbookmarks.

[0181] Once deduced bookmarks were distributed to viewers, the systemwould monitor their usage. If some minimal number of viewers jumped tothe given deduced bookmark but then shortly thereafter fast-forwarded ashort distance, or re-wound a short distance, this would be interpretedby the system as an attempt to adjust the bookmark. This adjustmentdistance would then be used to re-adjust the distributed bookmark goingto viewers for the first time, as well as used to re-adjust bookmarksthat were “in the field”, that is already distributed. Again, datacoming from viewers known to be “careful adjusters” would be given extraweight.

[0182] Certain viewers might be determined to have better skills indetermining accurate skip points. This might be determined by looking athow well their marks clustered around the average location for a givenmarkup point. The markup points offered by these users could then begiven extra weight in the overall averaging process.

[0183] The averaging process could take into account multipleinputs-viewers' fast-forward stopping points, viewer-generatedbookmarks, viewer-created segments that were saved, viewer adjustments,etc.

[0184] Aggregate User Feedback Used to Edit Breakpoints

[0185] Above we discussed how break points could be deduced by watchinguser's actions from which we could deduce breakpoints. Another way touse aggregated data is to watch how viewers use our proposed bookmarks.For instance, take the case of generating a break mark when the newsannouncer changes. This may not be the signal of a new story—it may justbe the anchor handing off to a field person for a report. We coulddeduce that by watching how early users of the metadata don't skip atthat break point, but watch the preceding section and go right onthrough this supposed next segment. If it were truly a break in thecontent, some percentage of people would be assumed to skip at thatpoint or close to it. Therefore, if a bookmark is not used by someminimal percentage of people (who have watched the entire previoussegment) as a launch pad to jump forward from, it would be assumed tonot be a meaningful break. If no one uses a bookmark, then by definitionit is not useful.

[0186] Correspondingly, if a high enough percentage of viewers skip outof the previous segment and then shortly skip out of this secondsegment, it could be assumed that the content is too similar and again,it is not a meaningful segment marker. Again, by definition, if there isan extremely high correlation between viewers skipping one section andthen the second, the two are probably very closely linked and probablythe same segment.

[0187] The exact percentages needed to make these decisions can beempirically tested as stated above.

[0188] But in general, the system can be organic and self-correcting.For instance, field data can always be used to second-guess a decisionmade by the system. If the system for instance, erases a marker, andthen sees monitored viewers starting to fast-forward through materialdemarcated by the old erased marker, it can re-insert the marker.

[0189] Combining Different Metadata Using Bayesian Statistics

[0190] The NLPE will not always accurately segment the show. As such, itwill be useful to combine this technique with others. Each techniquewill add additional information in determining the probability of abreak. For instance, scene change analysis will be used to deduce when ascene occurs. If such a change occurs close to where the close captionanalysis suggests there may be a topic change, then Bayesian statisticalmodeling will be used to predict the probability of a break.

[0191] Time-Based Data

[0192] Further data to add to the Bayesian analysis would include thetime duration since the last break. Each program could have stored afrequency distribution of how often a topic change occurred. As thetime-since-the-last-break increased towards, and then past, the averagelength of a segment, it increases the probability that an inference of atopic break is, in actuality, a real break. This time-based data wouldbe added to the data synthesized by the Bayesian tool.

[0193] In other words, it's unlikely that CNN News would have a10-second story. So the time-length factor would sharply mitigate theprobability of the system producing a break after ten seconds even ifthe closed caption text and scene change analysis suggest such. On theother hand, segments rarely go past 2 minutes. So as the length of asegment approached a long duration, the “benefit of the doubt” wouldstart to swing towards designating a break. Bayesian statistics is themethodology of revising probabilities based on new data.

[0194] Double-Indexing

[0195] Another goal of the system would be to develop two levels ofbookmarking—one equivalent to chapters and one equivalent to paragraphs.The methods discussed above could all be adapted to determining minorchanges from large ones.

[0196] By the same token, the system could produce “hard” bookmarks(ones it is confident in) and “soft” bookmarks. An interface could beoffered whereby viewers could be offered a choice of being very carefulin their surfing by jumping through all the soft bookmarks, or a bitmore relaxed and only deal with hard ones.

[0197] Training

[0198] The system could be trained to adaptively produce (better)auto-generated bookmarks, by looking, over time, for correlation betweenknown accurate segment markers (generated by hand or other accuratemeans) and those deduced though the means discussed above. The knowledgegained in this learning process would be used to update the Bayesianprobabilities.

[0199] System-Created Text

[0200] NLPEs can identify key words in the text. These could beassembled to form very cryptic slugs that would fit on a TV screen. Theymight form a sentence if there was room on the display (George Bush inChina), or the slug might just be a list of key words (Bush China). Thescreen display could be set up so that the user could hit the rightbutton once at a particular slug to see a sentence that scrolled off thescreen, or could hit a button to access a longer descriptive piece abouta story.

[0201] Abbreviations: The system would keep a large library of commonabbreviations and use these when needed in the slugs or otherdescriptive text to save space. This feature could be turned on or offby the user. Locating the cursor on an abbreviated word, and selectingit, would present the whole word. Alternatively, a viewer could go to anindex of abbreviations.

[0202] Auto-generated bookmarks could be created on a customized basisfor each viewer. Some of this computation could be done on the client orthe customization could occur by customizing the presentation ofbookmarks created by a central system.

[0203] Preference setting by each viewer would customize thepresentation in a number of ways. The user could input levels of“hardness”, the density of bookmarks, and the maximum or minimum lengthof segments desired. The viewer might also be aware of the level of“maturity” of bookmarks—that is the number of previous viewers uponwhich the deduced bookmarks are based. Have the bookmarks stopped“moving”? The viewer could also input keywords that would signify extrainterest.

[0204] Alternatively, the system could deduce these parameters (desireddensity, for instance) or keywords on a viewer-by-viewer basis. If auser continually skipped out of a segment shortly after landing eachtime, the system might deduce that the user was not that interested inthe content and therefore reduce the density of presented bookmarks.

[0205] If the system deduced a keyword for a user, it could then findthe closest bookmark with which to demarcate it. For instance, if akeyword is found, the system might lower its threshold of tolerance forcreating a bookmark thus allowing one to appear shortly before the word.In this manner if the user is surfing rapidly through the content, theywill be sure to catch that segment close to the relevant point. Thekeyword could be displayed on the screen as well. These bookmarkparameters could be displayed to viewers as they watched as visual iconson the periphery of the screen. In this way, viewers would be remindedof the information with which they could be navigating. It would alsoteach viewers what the unseen metadata was and encourage its use.Viewers would also be made aware of whether they were navigating withNLPE-derived markups or behavior-based deduced bookmarks.

[0206] User-Controlled Settings

[0207] Errors. An NLPE will never do a perfect job. It will sometimesgenerate markups that shouldn't be there (false positives) and at othertimes, miss breaks that do should be there (false negatives). Some usersmight have a preference for one or the other type of error. As such,viewers could have the option of setting a control that modulates theBayesian statistical analysis tool such that one type of error oranother was favored. (It is a bit of a zero-sum-game—trying to minimizeone type of error will increase occurrences of the other.)

[0208] User Selectable Lead Ins. Errors will also be made in finding theexact beginning and ends of segments. If the system tries to hit thespot exactly, it might often cut off some of pertinent material. Thiscould be annoying and make the viewer have to scroll back to find thetrue beginning. Consequently, and end-marking bookmark may be delayed toadd extra material to each identified just to “be on the long side”. Weenvision having the user be able to modulate this type of errortrade-off as well, as discussed above for marker error.

[0209] User Selectable Segment Types. The preferred system would letusers indicate preferences for certain segment types. For instance, asports fan might indicate a preference for jump balls (recognized by awhistle and characteristic picture composition), or applause lines onthe Letterman show. This preference information is more form-based thancontent-based, the usual parameter used for personalizing.

[0210] Using Bookmarks

[0211] Save Button. With this button, a user could take a segmentassociated with a bookmark (e.g., the programming between one bookmarkand the next) as stored at 151 and drop it into a “scrapbook” or vaultat 153. This scrapbook could have a specific amount of storage space at153 allocated to it. This storage space might be actual hard drive spaceon a viewer's PVR or shared storage on a network. In the case of theshared storage, the viewer's “ownership” of the stored content wouldindicated by metadata that associated that network-stored content withthat viewer. Items dropped into the space could be assigned permanent ortemporary storage. By default, segments would be sorted by, and assigneda label from, the show from which they came. The user could tag thesegment with additional tag via voice input, a keyboard, or by selectingfrom a menu. (Each show would have menus of clip-types that made sensefor that show—pass plays, tackles, runs, etc. for football). Thesegments in the scrapbook could be sorted by category bucket and orsegment type. In addition, a user-operated “Scissors” tool could be usedto clip off unwanted content. Playlists could be setup and segmentscould be sorted by type or by time, etc. Furthermore, the videoscrapbook could be implemented by having the users do both cuts todefine the segment. In this way, they won't rely on metadata toautomatically copy the piece out of the stream, although they could usethe metadata to navigate to the desired point at which to cut.

[0212] Scanning Playback. One playback tool that is not necessarilyassociated with automated markup may be thought of as a “scan mode,”similar to that found in radios. In radios, scan will jump from channelto channel, giving you a sample of each. In an equivalent PVR feature,the TV could play short segments of each segment and then go to the nextone if the viewer doesn't hit a Play button. This feature would be bestapplied where the chance of the user wanting to see a particular pieceis low and it's too tedious to keep hitting the Next button.

[0213] “Sweet Spot” Surfing. A NLPE would be able to find the heart of astory. Currently this technology is used to summarize a newspaperarticle for instance. In our application, the NLPE would identify thekey segment of a news story or other segment. This would be the “sweetspot” of a segment. Our editors could also demarcate these sweet spots.Alternatively, they could be deduced by watching where our viewers puttheir systems into slow motion, or replayed the content. Sweet spotsurfing could be a way to let users get right to the juicy part of eachsegment, without necessarily starting at the very beginning of thelogical segment. It could be the spot that a viewer jumped to whenhitting the Next button or employing the Scan button (see below).

[0214] Sweet spots could also be deduced by analyzing navigationpatterns produced by other viewers. These would be the portions ofsegments never fast-forwarded. In this model of sweet-spot surfing, thesystem would set the bookmark at the beginning of this section and letviewers land right in the middle of the larger section.

[0215] Segment Filters. The idea here is to treat ads or otherrepeatable segments of content as recordable scrapbook items. Thesesegments could be fingerprinted and have a duration associated withthem. When the PVR spotted an ad or other particular type of repeatedsegment, it would back up to the beginning of the segment and go to theend and demarcate the segment. These bookmarks could be loaded into thesystem for viewer use.

[0216] These segments could also be treated through a rules-based systemassuming that they are ads. For instance, the rules might say that anyad that the viewer has seen “X” number of times, be deleted, orautomatically skipped on playback, etc. In some cases, for instance withan ad-supported modality, the user might have to watch a segment of itbefore being allowed to continue on.

[0217] Features similar to Never Again and the Huntlist described in theabove-identified previously filed applications could be set up. TheNever Again feature is a personalized list of segments, which the userdoes not wish to see again. This list can be stored on the client or onthe network. The user could add a segment to the list during viewing bya command or could construct the list during a non-viewing session. TheHuntlist is a similar sort of list, personalized by viewer andconstructed in a similar manner. In this case, however, an item on theHuntlist would be given special status and highlighted in certain waysto bring it to the attention of the viewer. It could even beautomatically saved for that viewer. The user would go to a database andrequest the system pull down Budweiser ads. Our system would thendownload fingerprints for those ads to the client PVR to be used to huntfor the relevant clips.

[0218] Other ways to identify ads would be to look for scrolling text,800 numbers, and abrupt change in the frequency of scene changes. Inother cases, the “scene format” may suddenly go away. For instance, theFox scoreboard in the upper left, or part of the Bloomberg informationmatrix may go away to make room for the ad. If it were a baseball game,scene recognition techniques could use a database of shots of infields,hitters, etc. to detect that the game was still showing. If it is a newsshow, there may not be a talking head in the picture. A database ofnewscaster facial images or voices could be maintained for each show,and if someone not from the list is deemed to be present, this factcould indicate an ad. Similarly, a database of products commonlyadvertised, may be maintained and used to determine if advertising wasbeing viewed or not. In addition, an algorithm detecting the excitementlevel in a voice or other tonal quality might indicate it was not anewscaster, or even an interviewee. Ads might also have people speakingquicker. And currently, a lot of ads don't have closed caption text. Anynumber of these clues may be used in combination with Bayesiantechniques to determine the probability of an ad break.

[0219] Note, that some clues posit the location of a break point(switching of speakers, for instance) without knowing if there is achange in subject matter, whereas other clues indicate the presence ofan ad (mention of a product name, for instance) without knowing wherethe break is. By combining both types of information, the content may beboth segmented and categorized. This technique of combining contentinformation with segmenting clues may be used for other types of contentbesides ads.

[0220] Reminder System. An NLPE could also detect when a station waspromoting future items, segments, or stories. “Next week, we'll belooking at”, would be one example. When these were detected, the usercould be presented with an option of having a reminder sent to him towatch it. For live viewing, a reminder could be sent via email ordisplayed on the TV at some time before or during the playing of thissegment (or the show itself as the promotion by the station probablywouldn't specify the exact time during the show). Alternatively, thesystem could automatically record the upcoming show. It would bookmarkthe show segment if it could be located by text analysis and display areminder that it had been recorded and located on a program guide.

[0221] Sharing Locally Created or Edited Metadata With Other Users

[0222] Note that, when metadata is placed in an addressable location,other users may retrieve it on a peer-to-peer basis. In thisarrangement, a user might be simply supplied with a list of URLs atwhich other users having similar backgrounds, or viewers who were knownand trusted, could post reviewable metadata. In this way, a user couldaffirmatively recommend certain programming and affirmatively discourageother users from viewing other programming.

[0223] Similarly, one user could bookmark an individual program or asegment of a program, associate a recommendation or comment with thebookmarked content, and make the program or program segmentidentification data and the comment or recommendation available to aspecial interest group or to a specific individual. In order todistribute metadata to designated users, it may be structured to includeaddressee data which specifies individuals or groups, so thatbookmarking metadata of this kind can be affirmatively pushed totargeted users, or pulled by users who request metadata contributed fortheir specific attention, or for the attention of a group to which theybelong.

[0224] Using the facilities of an interactive digital cable televisionnetworks, a viewer could be watching a show live and want to recommendit to another friend. Using a remote control, the user could select oneor more friends from a preset displayed list and then transmit to thosedesignated persons a “watch this” message that might be displayed asclose-captioned text on the friend's screen. Properly programmed, thereceiver could provide the option to open a window on the TV screen fora PIP (“picture-in-picture”) display of the recommended show.Alternatively, using the Internet, a message could be sent via aninstant bookmarking messaging connection or by email to a designatedperson or persons.

[0225] Bookmarking metadata might also be transmitted to programmableVCRs and digital PVRs to automatically initiate the recording ofdesignated programs or program segments while, at the same time,advisory messages from the metadata sender to the target viewer could besent to notify the recipient that selected programming had been recordedfor their benefit. Such a system for remotely controlling a designatedreceiver would include a security firewall so that only authorizedsenders would be able to access and program the recorder. In addition,when recording space was limited on the target recorder, an appropriatealgorithm would be used to prioritize the importance of someone else'srecording suggestion so that important existing recordings were notoverwritten and space was conserved for future programming having ahigher priority.

[0226] Note that the use of such a “Watch This” bookmark messagingfacility is not limited to live broadcast material. Through the use of apredetermined program identification system, either based on asource-plus-broadcast-time identifier, or a unique program identifier,or a signature-plus-time-offset designation, someone viewing apreviously recorded program could also send a bookmarking message to oneor more persons recommending content which may (or may not) be availableto the recipient either in recorded form or in a future broadcast. Thereceiver of these recommendations would have the option to foregoprivacy and permit the message sender to access metadata (e.g. at 133 inFIG. 1) to determine in advance whether the recommended programming wasavailable to the recipient.

[0227] Metadata in the store 133 at the user's location which identifiespreviously recorded and locally available programming content (in one ormore of the content storage units 143, 147, 153 or 163) may betransmitted to the user data store 117 at the remote location to selectdescriptive metadata at 115 from the metadata store 113. In this way,recommendations, ratings, descriptive EPG data, and the like from bothprofessional reviewers and from other viewers may be returned to theuser as an aid to selecting programming content of interest from theavailable recorded materials.

[0228] When “watch this” messages of the type described above arerelayed to recipients via a remote resource, they may be combined toform aggregate recommendation data, enabling any viewer to identifyprogramming that has been most frequently recommended to others.

[0229] Although the present invention contemplates that metadata whichis created at one location and made available to another location andfurther that this metadata relates to broadcast programming content thatis independently available at both locations. Where appropriate, whencontent available at the location where the metadata is created is notalready available at a destination location, it may be transmitted withthe metadata. For example, locally created content (such as home videorecordings) may be stored at the user location, described by metadata,and both the content and the metadata may be distributed. In addition,program content providers may authorize the redistribution of theircontent under appropriate conditions (for example, under the conditionthat the advertising is not deleted), in which case both the content andthe metadata which was obtained from another source, or metadata createdlocally by a viewer, may be made available to other users. In onepreferred mode, metadata stored at 133 and published by a user through acentral server location or by a direct peer-to-peer connection mayinclude the URL or identifier of the program content which may beretrieved by another user who selects in by first displaying itsdescriptive metadata.

[0230] Storing Content at 102, 107, 143, 147, 153 and 163

[0231] As previously noted, content programming is initially stored at143 in a mass storage unit that may also serve as a circular bufferstore to enable the user to pause, replay, and fast forward within apredetermined duration of recently received incoming broadcastprogramming. By using the local edit controls at 136, the user canemploy metadata to identify selected program segments for inclusion in aprogram playback library. Metadata created by the user or the remoteserver, or signal processing techniques, may be used to parse theprogram content as broadcast into segments at 145, and further metadatamay be used to select, discard or modify the programming content at 151,161 and 171 before it is reproduced.

[0232] It should be noted that the storage units shown at 143, 147, 153and 163 in FIG. 1 are “logical” storage units, which can be, andnormally should be, implemented by single physical device or group ofdevices. During use, actual copies of program segments need not becopied from one logical storage location to another. Instead, thecontents of the logical or “virtual” storage units 147, 153 and 163 maybe defined by metadata which describes and provides pointers two programsegments which are a subset of the program content stored in the “inboxstorage” unit 143. Only some of the parsed program segments identifiedby metadata in the logical parsed segment storage unit 147 are selectedfor potential playback by the user and placed in logical storage unit153, and the logical contents of the virtual storage unit 153 may bemodified (by the addition or deletion of advertising, for example) andplaced in the virtual storage unit 163. When program data is logicallytransferred form storage unit 143 to 147, and then from unit 147 to unit153, it becomes increasingly insulated from destruction. In general,content, which has not been selected by the parsing process at 145, iseligible to be overwritten, as are parsed program segments, which are“discarded” at step 151. In fact, unparsed and discarded content is notactually overwritten until additional space is required to store newincoming broadcast content.

[0233] Note also that, as long as program content contiguous to anyprogram segment stored in virtual storage units 147, 153 and 163 has notbeen overwritten, the start and end boundaries of these virtually storedsegments may be modified since the location and extent of each suchprogram segment is defined by metadata and not by separate storage. In asimilar fashion, the storage unit 107 at the server is preferably avirtual store whose logical contents are defined by metadata, whichspecifies programming content stored in the input store 103. Asdescribed later in connection with playlists, if a user is viewing orlistening to a given program segment, he or she may issue command to“continue playing” a given segment beyond its end boundary, enabling thecontext of a segment to be reviewed. In the same way, content whichcontinues to be available in the “inbox” storage 143 before the startboundary indicated for the program segment may be reviewed as well onrequest.

[0234] The system may capture more than one copy of a given programsegment (e.g. song) if desired. Multiple copies may be compared in orderto create a better single copy. For example, two or more copies can be“averaged” to accentuate shared components and suppress noise ortalkover components existing on only one recording. After duplicates areprocessed, the extra copies may be discarded to save storage space. Inaddition, editing facilities at 135 permit the editor to readily comparethe two copies of a given program segment select the better of the twofor inclusion in the program library. The user may also rate the qualityof a particular program segment, or the quality of programming availablefrom a particular source, to facilitate the elimination of lessdesirable duplicates.

[0235] When Community Markup (CM) is used to enable users to sharemetadata and program quality ratings, an automatic search may beperformed for best of several copies. With a community markup scheme,metadata, including song quality, is stored at the central server at113, or shared among users on a peer-to-peer basis. For example, suchcommunity-generated metadata may accordingly be used to determine if thequality of a new copy is better than a previously recorded copy in apersonal library. Thus, a previously recorded copy having a reducedrating due to DJ talkover could be automatically replaced by a recentlyreceived higher rated copy.

[0236] A user might collect hundreds of program segments with just a fewdays of recording of broadcast materials. Cleaning out the “inboxstorage” at 143 and selectively deleting unwanted songs, even if thereare duplicates would require a prohibitive effort by the user.Therefore, any method of helping the user make a decision quickly aboutthe desirability of a song would help this process. To this end, theserver could make available thumbnail summaries or short snippets ofeach program segment identified in the database, or the metadata couldinclude a designation of a snippet, and these snippits could then bepresented to the user to facilitate the save/delete selection process.

[0237] The availability of snippet identification data also enables theuser to more quickly scan the program segments or songs available in thelibrary. These snippets may be presented in succession to the user, in afashion similar to the manner in which a car radio scans sequentiallyscans from station to station until interrupted. Because each snippetidentifies the readily recognizable “sweet spot” of each programsegment, the program segments may be readily identified by the user. Theuser may also manually scan from snippet to snippet by pressing a “Next”button when hands-free scanning is not desired. Either way, the user canuse the recorded snippets to more readily select desired programsegments during playback, or to skim through the recently depositedprogram segments which have been initially parsed at 145 to make quickmanual select/discard decisions at 151 regarding the desirability ofkeeping a given program segment.

[0238] Business Models

[0239] The creation and distribution of metadata relating to broadcastprogramming may be sponsored by a variety of business models.

[0240] The metadata may be distributed on a subscription basis, witheach user paying a fee to the metadata provider. The use of metadata maybe accompanied by the presentation of advertising presented along withthe programming content either by modifying the content as illustratedat 161 in FIG. 1. Advertising revenue may also be derived from thepresentation of promotional material when program guide information isdisplayed, or when editing screens are displayed that enable users togenerate comments or program recommendations. “Public” authors ofmetadata, which is shared with others, could be compensated on acommission basis, or could receive discounts from subscription fees inreturn for supplying metadata to others. For example, metadata createdby a markup editor could be compensated with a pro-rated share ofrevenues based on how many of his or her metadata items were usedcompared to the total system-wide use of metadata.

[0241] Individual users may be compensated for watching advertising, andthis compensation may take the form of a reduction in subscription feesor an actual payment. Note that, by using user preference data to directadvertising to those who would have the greatest interest in thespecific service or product advertised, both the advertiser and the userare better served. Advertisers reach those potential customers havingthe greatest potential interest in the advertised material, and usersneed not be burdened with the presentation of advertising in which theyhave no interest.

[0242] When advertising that is provided as part of the contentprogramming, or inserted into the content as noted above, the user maypress an “information” button (normally used to trigger a displaydescribing the program currently being played) to obtain additionalinformation about the advertised product. In this way, the useridentifies products and services about which he or she has a particularinterest, and the advertiser is able to provide information (includingthe URL of an Internet resource containing detailed information), whichwould otherwise be unavailable to the user.

[0243] A remote control device may be used to accept positive ornegative rating metadata during a program without interrupting theprogram. For example, the viewer may press a positive rating button onthe remote control device one or more times to signal a level ofrecommendation, or press a negative rating button one or more times tosignal a level of disapproval.

[0244] When previously parsed programming segments are being played tothe user, the user can issue a request to insert a comment or annotationat any time during a segment. Then, at the end of the segments, adisplay screen or other prompt will appear. In this way, the playbackunit can accept a comment, annotation, rating, or the like at the end ofthe segment. If live broadcasting is being viewed, the incomingbroadcast can be recorded so that viewing can then be resumed on atime-shifted basis after the metadata is created.

[0245] During playback of segments having associated metadata, theviewer may select the manner in which the metadata is presented. Forexample, descriptive comments may be displayed as close-captioned textor in a separate screen window without interrupting the program display,or the metadata may be displayed at the beginning to aid the view indetermining whether or not to watch the program or program segment aboutto be displayed.

[0246] The technique of broadcasting programs (content) for storage atthe user's location and thereafter performing time-shifted playbackunder metadata control may be used as a primary system of programdistribution by content owners (record companies, broadcasting networksand stations, etc.).

[0247] For example, radio or television programming could be broadcastin compressed and/or encrypted form for local storage, a “recordselection” file of metadata may then be used to selective recordprogramming of probable interest to the user at the user location, andthe resulting programming may then be selected for inclusion in theuser's program library and selected for playback under user control asdescribed herein. The cost of programming could be financially supportedin whole or in part by subscription fees, or by advertising, and userscould elect the extent to which they were willing to view advertising inexchange for reduced subscription fees. Advertising segments, likeprogramming content, may be inserted into the programming at playbacktime and selected based on user preferences and demographics, helping toinsure that the advertising presented is relevant to the consumer andhence of more value to the advertiser and the consumer. With the consentof the copyright owners, radio and television broadcasts would enableusers to purchase music singles, entire CDs, music videos and completemovies for inclusion in their personal radio and/or television programlibrary. Program catalogs distributed in advance of the broadcast couldbe used to alert the user to future broadcast programming that could berecorded under metadata control at the time of broadcast distribution.

[0248] With or without pre-published catalogs and playlists, contentowners and broadcasters could use a content watermarking system, oridentification codes imbedded in the program as broadcast (for example,using the RDS standard) to make it possible to identify program segmentsregardless of the time and frequency of the broadcast. In this way,watermarking and identification code systems used for other purposes(e.g. broadcast monitoring for royalty verification, or to preventillegal copying) could be used for the additional purpose of controllingrecording and playback by licensed users under metadata control.Identification codes or watermark patterns may be included combined inhuntlist and playlist metadata data with additional metadata describingthe identified programs.

[0249] To promote particular songs, albums, subscription or freebroadcast programming, record companies and broadcasters mightdistribute metadata, which presented selected segments of recordedprogramming in organized “previews” designed to promote individualprogram segments. Sample songs and programs might be made available freeof charge to promote related programming, in which case mechanisms couldbe included minimize any possible cannibalization of program sales, butwhile providing introductory exposure to new programming. As such, thefollowing features might be implemented:

[0250] 1. Restricting how many songs off an individual album, or howmany programs from serialized programming, could be captured.

[0251] 2. Limit the duration of any preview segment.

[0252] 3. Limiting the size of the “preview” library.

[0253] 4. Limit the “life” (duration) that a program may be previewed,or limit the number of times a program segment (e.g. audio song or musicvideo) can be played before it must be purchased or paid for on a usebasis.

[0254] 5. Charging a subscription fee for the right to view previewcopies

[0255] 6. Prevent preview segments from being transferred to anotheruser.

[0256] 7. Inserting advertising into previews.

[0257] Selecting and Modifying Content at the User Location

[0258] As indicated at 143-171 in FIG. 1, the metadata provided by theremote facility, or by other users, or by the local user, may controlthe selective recording and rearrangement of program segments to formcomposite programming. Thus, segments extracted from several newsprograms might be recorded during the day with all programming dealingwith a particular topic being consolidated into a single compositeprogram devoted to that topic. The selection of the components of such acomposite program may be made at the remote location by using thepreference data supplied by an individual user stored at 117, or may bedone at the user location by matching locally stored preference dataagainst program description metadata received from the other locationsor locally produced descriptive metadata. Note also that such acomposite program need not be constructed in advance at 161 and storedat 163 for playback, but may instead be assembled “on-the-fly” simply byselecting identified segments in the proper order from the segment store147.

[0259] Selecting Segments for Playback at the User Location

[0260] As noted earlier, the metadata, which is available to the user,may include electronic program guide (EPG) data for displaying a listingor matrix of available programming, including both live programming andrecorded programming. The user may select items from this EPG display torecord or play incoming broadcasts (or both), may play previouslyrecorded programming, or may identify future programming to be recorded.

[0261] During playback of recorded material, and during the recording ofnew material, a progress bar that shows the location within a programthat is currently being viewed can be displayed at the user's requests,typically occupying only a portion of the screen while the video contentoccupies the remainder. Segment markers can be noted on the bar andassociated with icons to indicate the presence of descriptive metadata.Using a mouse or remote control to “click on” or select a segmentdisplayed on the content bar would then alternatively cause the metadataassociated with that segment to be displayed, or would resume playbackof content at the beginning of the selected segment. Segments as shownon the progress bar could be color coded based on a program rating toenable the user to quickly view highly rated segments, or to skip lowerrated segments.

[0262] In addition, metadata about segment quality or other attributesmay be displayed on the screen using suggestive icons (smiling faces,frowning faces, etc.) while a segment is being shown helping viewers tomore quickly decide whether to hit the “next segment” button or achannel surfing button on a remote control unit. Icons indicating theavailability of additional descriptive metadata may also be displayed onthe progress bar, or associated with programs listed in a displayedprogram guide.

[0263] Because metadata may exist in many forms from many sources, theuser may be given the opportunity to enter display preferences thatcontrol the manner in which metadata is displayed. Thus, metadata fromespecially trusted sources may be preempt regular programming and beprovided with use of the entire screens, while other metadata may bedisplayed as closed captioned text or as icons, or without any displayunless the view specifically requests the presentation of metadata for aparticular program segment.

[0264] One of the most important mechanisms for assisting a user inlocating desirable programming is the use of metadata to enhance thecontent and operation of the electronic program guide. Metadataindicating a user's preferences which is derived from both thepreferences directly expressed by the viewer and by preferences inferredfrom the user's viewing and metadata creation activities may be used toselectively display and highlight particular programs in the programguide listing. Icons or highlighting may be used to identify listedprograms and segments for which additional metadata is available fordisplay to the user upon request. Metadata which ranks programs may bedisplayed using rating icons, color coding, or highlighting to guide theviewer toward higher rated programs.

[0265] Note that program guides may display listing of previouslybroadcast materials which are available in local storage, broadcastprogramming which will be available currently and in the future forviewing and recording, and “content on demand” programming which existsas retrievable resources on program servers and on storage maintained byother users and shared on a peer-to-peer basis with other users.Metadata describing all such programming content may be located using anelectronic program guide format which permits the extensible display ofadditional metadata and the selection of particular program content forviewing and recording.

[0266] Playlists

[0267] The metadata, which describes individual program segments, may becombined to form an ordered playlist. As described in detail in U.S.Pat. Nos. 5,271,811, 5,732,216, and 6,199,076, and in co-pendingapplication Ser. No. 09/782,546 filed on Feb. 13, 2001, by James D.Logan et al., by James D. Logan et al., the metadata as assembled at theserver and transmitted to the user location may take the form of aplaylist consisting of a scheduling file of metadata which specifies thecontent and schedules a default playback sequence in which that contentis reproduced. At the user station, the scheduling file may bereorganized to alter the content and schedule of a playback session. Asdescribed in the foregoing patents and application, the content of theplaylist may be varied in accordance with preferences associated witheach user.

[0268] The metadata stored at 133 is available to the user to facilitatethe selection and navigation of available program materials. Themetadata may include a playlist that specifies the sequence within whichprogram segments will be played back. Navigation controls including skipforward and skip backward controls may be used to skip the remainder ofthe segment being played and resume the playback at the beginning of thenext segment, at the beginning of the current segment, or the beginningof the prior segment.

[0269] In addition, the user can vary the playback speed, requestcompressed playback where periods of silence and/or unchanging imagesare skipped. Playback speed can be automatically increased (both speechand video) under metadata control or by analysis of the content when theaction is minimal (e.g. huddles in a football game) and slow down whenthe action picks up (e.g., after the football is put in play). Sequencesthat are candidates for rapid replay may be specified by metadata or canbe determined by identifying programming when no voices are present inthe audio component and minimal changes are occurring in the videoimage. Automatic or manual playback speed adjustments may appliedindependently to different program streams displayed concurrently in asplit screen or in a picture-in-picture (PIP) display. As notedelsewhere, a viewer's decision to skip, speed up or slow the display ofa particular segment may be used in a segment rating system as voteindicating that viewer's level of interest in that segment. In addition,such viewer actions can be used as an indication of the viewer's subjectpreference or disinterest in the subject matter of that segment and suchdecisions, on a cumulative basis, can be used to develop a preferenceprofile used to automatically recommend or select programming forrecording or playback.

[0270] Metadata may also be employed to specify a play list composed ofextracts from a stored program, enabling users to view a preprogrammedpreview of a given program, or to view a shortened summary of theprogram instead of the entirety of the program. When passages ofparticular interest to the viewer are presented, the user may be giventhe option of viewing that segment in its full context, and then switchback to the shortened version on demand. Note that a decision by aviewer to switch to the full context for a particular segment presentedin a preview or summary may be taken into account as a positive ratingfor that segment, or as a preference indication attributable to thatviewer. As noted elsewhere, the “snippets” which are viewed or listenedto during playlist navigation or scanning may a highlight or “sweetspot” of the program segment which is designated by metadata.

[0271] Metadata labels may be displayed in a list, or as subtitles, toassist the user in rapidly locating desired segments for playback. Amosaic of images, each selected from a single segment, may be displayedas a visual cue to assist the viewer in locating a desired segment froma sequence of segments. When the metadata includes descriptive text,keyword searches can be performed to identify segments described withmatching words.

[0272] The presentation of probable preferences, whether by icondisplay, highlighting particular items on a program guide listing, orthe like, may be based on either a local or remote analysis comparisonof the user's preference profile with the metadata that describes thesegment's program content. If done remotely, icons and highlight controlmetadata may be sent with the program guide or programming material todirectly control the user's display whereas, if the analysis isperformed locally by comparing locally stored preference data with thedescriptive metadata, the user's privacy may be better protected sincepreference information need not be transmitted outside the user'slocation.

[0273] The subdivision of program materials into logical segments makesit possible for a viewer to save individual program segments, and theirassociated descriptive metadata, into a virtual scrapbook consisting ofsegments tagged as saved for later viewing. If desired, a viewer mayedit such a scrapbook play list to delete or crop particular segments,rearrange the sequence in which they are to be played back, and addannotations or comments. The resulting program sequence can then bepersistently stored in the available local storage area, or transmittedas message containing both content and metadata to another user. Theuser may also be provided with the ability to record the fact that aparticular program segments was found to be particularly interesting orenjoyable, thereby affirmatively recording a preference for furtherinstallments of the program and/or for other future or recorded programshaving similar content. Note that the act of saving a given program orprogram segment into the user's scrapbook may be recorded as positivevote in that program or segment's approval ranking, and as an indicationof the subject matter interests of the viewer.

[0274] Also, while viewing a particular program segment, the user may begiven the option of deleting that segment from storage, deleting anentire program sequence to which that program belongs, or affirmativelyrecording dislike for programs of that type which can be taken intoaccount when the preference makeup for that user is employed toautomatically select, recommend, or discard different programs andprogram segments.

[0275] In order to create truly personalized radio or television, it isdesirable to create playlists on a continuing basis. In addition, it isdesirable to allow users to create their own playlists and to randomizethese playlists to automate the sequence in which selected programsegments are played, or to automatically play those segments in a randomsequence.

[0276] The central server, or the local system, may generate playlistsbased on a combination of shared and personal data. The shared data mayidentify program segments (e.g. songs or informational segments), whichgo together and further indicate a preferred playback sequence forassociated segments. The personal data may be based on the user data(locally available or uploaded to the server and stored at 117), whichmay identify which program segments are available to the user and whichsegments have been previously played, and when. This shared and personaldata is then processed to produce a recommended personalized playlist,which is made available to automate the user's playback sessions.

[0277] In order to have the server-based playlist generation mechanismwork well, it needs to know as much as possible about the user'sdemographics, expressed preferences, listening habits and experiences.As described in U.S. Pat. Nos. 5,271,811, 5,732,216, and 6,199,076, thecontent of the scheduling metadata (playlist) may determined from theweighted combination of the user's demographic characteristics,expressed subject matter preferences (e.g. particular music genres andartists), and “log file” data which identifies what, when and how theuser previously played. At the time of playback, the user may alsospecify a positive or negative rating to the segment being played. (Seealso, the discussion of the negative rating “Never again” button and thepositive rating “huntlist”) discussed separately.

[0278] In addition, to tracking recorded program segments actuallyplayed, the log file may advantageously record the identity of livebroadcast programming or programming from physical media or othersources (e.g. music or movies on compact disk, downloaded MP3 files,etc.) which is viewed or listened to.

[0279] The user may designate a preferred session length for theplaylist. Thus, in a radio based system for use by commuters, thesession length may be related to the average transit time to or fromwork. Only those program segments having the highest positive weightedrating and which, together with other high rated program, have ancombined playing time that approximates the requested session length areincluded in the playlist.

[0280] Playlists, program guide data, and compilations from othersources may be aggregated and presented to the user. For example, theserver or the user location may employ a searchable database ofavailable program materials (e.g. songs, albums, movies, etc.), whichthe user may use to select a list of desired programming. This desiredprogram list may then be used to form a huntlist for broadcastprogramming saved to the inbox storage at 143, to transmit to the serveras user preference information, or to provide an indication of theuser's subject matter preferences which can be used identify relatedavailable programming which the user may wish to view or listen to.

[0281] Additional implicit metadata regarding user preferences by may bederived by monitoring other forms of users' interaction with music. Whenthe user change broadcast stations when viewing live broadcasting, skipsa song being played from a playlist, or rejects particular programsegments from a server produced catalog of available programming, anegative rating for that program segment can be inferred. Conversely,when the user plays a song or program segment multiple times, requestsadditional information about a program segment or artist, etc., thisbehavior can be interpreted as a positive indication for that program orsubject. Song preference metadata of all types will be used with ourSelector-type program to optimize the construction of a personal radiostation. Alternatively, the data may be ported over and used by asubscription music service.

[0282] The user interface presented to a user for program library andplaylist management may be designed using the interface for an emailclient as a metaphor. Just as email is “pushed” at the user and thensorted, read, and filed, a playlist manager presents a list of programsegments that are available in the user's personal library. Programs,which have previously been played, may be identified by a distinctivetype font or color. Once listened to, the style in which the programsegment is listed is changed. Users may sort the program listing list byartist, program name, date and time of capture, source (e.g. radiostation call letters), recording quality, user rating, and otherparameters. Multiple sort fields will be allowed; for example, thelisting could be sorted by source first, and then by time of capture.Any program on the list may be selected (by clicking or by entering itslist number). When selected, a given program listing may be immediatelyplayed in its entirety, a “sweet spot snippet” only may be played as apreview, or the selected segment may be added to a playlist, or moved toa user-created and user-named “folder,” or to a system -created folder.

[0283] Metadata which indicates the subject matter category orcategories (or genre) to which a program segment belongs, or indicatesthe artist name, album name, or series name may be used to create aninitial set of system-created folders and sub-folders for programsegments. Users may move individual program segments to and from thesefolders and user created folders. When a request to delete a programsegment in one folder is issued, and the same program segment alsoexists in other folders, the user may be given the option of deletingthat program segments from all folders at once. A “trash” folder mayhold a listing of all deleted program segments which are, in fact,retained in storage until the contents of the “trash” folder are deletedin whole or in part (in the same manner that a “trash folder” in manyemail clients operates). Category indicating icons, different fontcolors and styles, etc. may be used on any single listing to assist theuser in visually selecting particular program segments.

[0284] Program segments identified in a playlist or folder may be sharedwith others; that is, the metadata may be transmitted to others forinclusion in their huntlists or program library if the underlyingmaterial is already available, along with a covering “forwarding memo”from the sender. Metadata may be transferred in a standard format as aMIME attachment to email, or may be shared using other forms ofpeer-to-peer transfer.

[0285] An application particularly suitable to video, the user mightprefer an alternative to working off of a list, which could require theuser to go back and forth between the list and the video screen. In thiscase, program listings and selection menus may be superimposed over theimage, or in window or frame adjacent to the viewing area. Visualprompts, which characterize the currently viewed programming, may alsobe displayed. For instance, short descriptions of the program segment, arating value, an indication of the source and time of the originalbroadcast, or any other information derived from the metadata may bedisplayed concurrently with the program.

[0286] Navigation cues can be displayed, such as a “forward arrow” shownon the screen during a low rated segment, encouraging the user to skipto the next program on the playlist. For video or audio, the user mightwant to have more data available at once and so multiple icons may beshown on the screen at once. By skipping to last of four forward arrows,the user might jump ahead four segments. By the form or color of theicons may indicate each corresponding segment's rating or subjectmatter, permitting the user to more easily directly to a desiredsegment. A linear map of the content of the playlist, or of thecurrently playing program segment, might be presented across the bottomof the screen, letting the user go right to a desired segment or scene.The user may also be given the ability to modify the playlist byhighlighting programming to be skipped. In addition to skipping ordeleting segments (negative actions) another user interaction could beto “seek” or find similar segments immediately, or a request to includesuch similar segments on the user's huntlist. In addition, the user mayelect to “continue” playing a given segment beyond the end boundaryindicated by the metadata for that segment, or to review programmaterial broadcast before the start boundary for that segment.

[0287] With a handheld device, the user could hold the skip button downfor an extended time and that action would delete or skip over allsegments similar or associated with that segment. Attributes could be onthe screen that the user could click that would preface the secondaction of deleting or finding etc. In addition to deletingnon-qualifying segments, the system could rearrange segments. Again,this could be done with varying levels of user involvement, reaffirmingthe feature each time or setting a preference for rearrangement once. Ina more automatic system, the user could re-select preferences eachsession. The segments would be compared against these preferences andnon-qualifying segments cancelled. More automatic, would just be areaffirmation of existing preferences.

[0288] Playlist control is not limited to specifying recorded segments.For example, a commuter may wish to listen to a program of recordedmusic from a playlist, but further specify that the recorded playback isto be interrupted at predetermined times to permit the user to listen toa favorite scheduled news, weather or traffic report as a livebroadcast. In this case, the playlist includes the designation of bothlive and recorded programming and dynamically alters the playlist sothat live programming can be played at its broadcast time. Because theplaylist can control the tuner or tuners, the user can be presented witha hands-free combination of selected live and pre-recorded programming.To assemble such a playlist, the metadata provided from the serverincludes program guide data for future programming. To the extent suchprogramming is serialized, the user may request that a given liveprogram segment be dynamically included in any playlist, thuseffectively “interrupting” scheduled recorded programming to bring theuser each pre-selected live broadcast.

[0289] Advertising Preservation

[0290] It may be important to prevent the user from skipping theadvertising, which provides financial support to the broadcaster andothers. To this end, segment start and end marks may be placed in such away that the advertising, which supports a segment, is always includedin the segment. The advertising could also be placed in a “skipprotection” zone. For example, if a program segment was supported byadvertising content at both the beginning and end of the segment, anattempt to skip the advertising played at the beginning of the segmentwould cause the player to skip to the beginning of the next segment(thus preventing the user from playing the content without firstlistening to the leading advertising segment. Any attempt to skip theadvertising at the end of the segment would simply be ignored.

[0291] In the community markup system, editors contributing markup wouldbe prohibited from placing skip marks in a fashion that would permitadvertising to be skipped without also skipping the supported content.

[0292] Selecting Program Sources

[0293] A user typically has many different broadcast stations to choosefrom, and recording all broadcasts from all stations would exhaust, orat least misuse, local system resources. Accordingly, it is desirable toprovide mechanisms for automating or assisting the user in the selectionof program sources to record. Typically, this selection is made based onboth frequency and time; that is, selecting different frequency channelsat different times to capture programming of the greatest interest tothe user.

[0294] Because the server typically has a database which indicates whatprogram segments were broadcast by what stations over a time interval ofrecent days or weeks, and because the server further has available to itinformation from the user containing user preference information,including requests for particular programs, for particular series ofprograms, for particular subject matter categories or genres, forparticular artists, and the like, it is possible to match the userpreferences against the broadcast histories of those sources which areaccessible to the user and provide the user with data indicating whichstations are most likely to broadcast subject matter of interest to theuser, and a what times. Since the user typically has only one or a fewtuners available, and/or a limited capability to record plural programsat the same time, and limited storage space, predictive tuning may beapplied to increase the likelihood that programming will be capturedwhich best fits the user's preferences.

[0295] Thus, the user's huntlist of desired programs, together withpreference information which is expressly provided by the user orimplied for the user's prior selection history, is used to develop arecording schedule file which identifies particular stations and thetimes of day when those stations typically broadcast programming ofinterest to that viewer. This recording scheduling file of metadata isthen downloaded from the server to the client location and used tocontrol the selection of program material received and stored in theinbox storage 143. Alternatively, the recording schedule file may beused to recommend a recording schedule to the user, thus alerting theuser to desirable recording opportunities that might otherwise beoverlooked. For example, the recording schedule file may be used tohighlight sections of an electronic program guide to assist the user inmaking informed selections of particular programs of interest. When arecording selection file is used to automate the recording process(typically without requiring the attention of the user), the content ofthat file may be transmitted to the server so that the server can thenreturn a program segment identification file which identifies thecontent of the programming recorded, and permits individual recordedsegments to be selected by the user, or automatically compared with theuser's huntlist to select desired programming for inclusion in theuser's library.

[0296] The “record selection file” may also be used to provide “handsfree” automated tuning of live radio broadcasts. The record selectionfile developed based on the broadcast history of available stations andon the user's preferences may be used to supplement a listingidentifying those programs which are specifically requested, with theresult that the system makes an informed “guess” about the station mostlikely to broadcast live programming of interest. Normally, however, itwill be preferable to allow the system to record programming in advanceto create a library of program segments which are of know interest tothe user, and then to insert these programs on a time shifted basisbetween specific live broadcasts identified by the user.

[0297] Live and time shifted broadcast programming may be organized bygenre, permitting the user to select a specific type of programming. Forexample, the station selection push buttons in a car radio might beassociated with different kinds of programming: a “news” button wouldplay the most recent news broadcast from the beginning on a time shiftedbasis; a “traffic” button would play the most recent traffic report; and“classical,” “jazz” and “alternative” buttons would play back recentlyrecorded music in each specified genre. The user would pre-assign thekind of programming desired, and a “record selection” list calculated tocapture desired programming in each category would be provided to theuser. Programming which is time sensitive (traffic reports having a highpriority, news programming a medium priority, and music having a lowpriority) would take precedence to insure that recently broadcastinformation is always available.

[0298] A Personal Video Recorder Implementation

[0299]FIG. 2 of the drawings shows the principal data flow paths in apreferred implementation of the present invention which extends thecapabilities of a personal video recorder “PVR” (also called a digitalvideo recorder “DVR”). The invention may also be implemented using aNetwork Personal Video Recorder (NPVR) in which video programming isstored at a network node and fed to the client device on demand, withpause, replay, fast-forwarding controls being provided to the viewersimulating the capabilities provided using a PVR's local storage. Asshown in FIG. 2, the PVR performs the data storage and manipulationfunctions shown at the left of the vertical dashed line 201. Remote datastorage and manipulation services are performed as shown in the middleof FIG. 2 between the vertical line 201 and a second vertical line 203.These services may also be accessed by additional client devices asshown at the right of the dashed line 203.

[0300] The PVR includes a processor for executing programs whichperforming data storage retrieval and for controlling the display,recording and playback of video programming using integrated electronicprogram guides. See, for example, the ReplayTV PVR described in U.S.Pat. No. 6,324,338 issued Nov. 27, 2001 entitled “Video data recorderwith integrated channel guides” and the TiVo PVR described in U.S. Pat.No. 6,215,526 issued on Apr. 10, 2001 entitled “Analog video tagging andencoding system.” Controls for pausing, replaying, and fast-forwardingtime-shifted television programming stored in a digital circular bufferare described in U.S. Reissue Patent 36,801 issued on Aug. 1, 2000entitled “Time-delayed video system using concurrent recording andplayback.” In each of these arrangements, a programmed processorcontrols the recording and playback of video programming which is storedin digital form on a conventional hard disk drive. PVRs are increasinglybeing incorporated into the set-top-boxes (channel converters) providedby satellite and cable programming providers and utilize the electronicprogram guide provided by those services. The functional equivalent ofthe local program storage provided by a PVR may be achieved by video ondemand (VOD) services which store program material at network nodes nearsubscriber sets and download selected programming over a broadbandconnection upon the request of the viewer or in anticipation of theviewer's probable future program selections.

[0301] As described in more detail below, the PVR is provided withcontrols that may be manipulated by the viewer, typically in the form ofa remote control device coupled to the PVR by a wireless or infraredcommunications link. These interface controls typically operate inconjunction with the television screen display which provides menus,prompts and other visual displays to aid the viewer in performing threetypes of control functions: playback control as seen at 211, recordingcontrol as shown at 213 and content navigation as indicated at 215.

[0302] The playback control 211 controls the playback of stored videoprogramming seen at 217, stored electronic program guide (EPG) data seenat 218, application data such as standard templates stored at 220,metadata describing programs and program segments stored at 221, andother system control data stored at 222.

[0303] The recording control 213 permits the viewer to utilize the EPGdata 218 and the metadata 221 to control the recording of availablecontent in the local video store at 217.

[0304] The PVR further includes storage at 225 for storing executableapplication program code and storage at 227 for maintenance information,usage data, etc.

[0305] The locally stored video and EPG data seen at 217 and 218respectively in FIG. 2 are supplied from the live video source 230,typically a connection to a satellite or cable television provider(“MSO”), a conventional broadcast tuner, or some other video programmingsource indicated 231 which supplies programming and program guidecontent 232.

[0306] The application data 220, the EPG data 218, the metadata storedat 221, and the system data 222 may be downloaded via a data connectionseen at 240, which may be the same physical communications link thatsupplies the live video feed 230, or a separate data link such as anInternet or dialup connection. The data download feed 240 is may beprovided in part by a PVR support service which supplies EPG data 241from an available EPG database 242 and a database for storingprogramming suggestions 243 provided by the PVR service. In addition,the PVR service may provide information such as the “To Do List” 244 toassist the viewer to perform certain tasks, and to control the PVR'sperformance of automated tasks such as recording selected programsidentified by the PVR service.

[0307] Metadata stored at 221 in the PVR, and additional EPG,application and system data stored in the PVR at 218, 220 and 222 may bedownloaded via the download connection 240 from the metadata serviceprovider which provides a metadata update facility at 250. This facilityis coupled to a file server 251 and/or a database server 252 for storingmetadata including data describing individual program segments,playlists, huntlists, etc.

[0308] In addition, metadata contributed by other users and stored in apublic database 254 as well as private metadata from a database 255 maybe downloaded to the PVR by the metadata service's download facility250. The metadata stored at 251-255 may be created, edited and deletedusing a Web server 261 or other server 262 operated by the metadataservice to permit the public (other viewers) to contribute to themetadata as illustrated at 264, as well as providing the ability foremployees of the metadata service to create and modify the storedmetadata as indicated at 265. Using the PVRs data upload facility seenat 250, usage data, “watch this” selections, playlists, and otheruser-generated metadata may be uploaded from the PVR via the user datasubmission facility 272 provided by the metadata service which maysupply both the public and private metadata stores 254 and 255.

[0309] The executable program code stored in the PVR at 225 and themaintenance data stored at 227 may be updated through a maintenance datalink 280 which receives program code and maintenance data downloadedfrom either the PVR service at 245 or from the metadata service provideras seen at 290.

[0310] The video program storage seen at 217 in FIG. 1 may be subdividedinto virtual storage areas corresponding to the storage 143, 147, 153and 163 described above in connection with FIG. 1. The metadata storageshown at 221 in FIG. 2 corresponds to the metadata storage shown at 133in FIG. 1. Although separate storage areas are shown in FIG. 2 forstoring application data, EPG data, video date, metadata, executablecode, etc., it will be understood that all such data may be persistentlystored on the PVRs hard disk from which it is loaded into RAM storagefor use as needed.

[0311] The Viewer Interface

[0312] The interface presented to the viewer by the extended-capabilityPVR shown in FIG. 2 preferably displays an on-screen program segmentindex of the type shown at 310 in FIG. 3. The index 310 transparentlyoverlays the content of the normal programming whenever a “segmentguide” button is pressed on the remote control unit, or selected from amenu of other options. Alternatively, when the segment guide islaunched, the portion of the display showing content is shrunk as shownat 405 in FIG. 4, providing room at the right for an index list ofsegment labels at 410 and an “information pane at 412 below the contentwindow 405.

[0313] The segment index displayed is normally a subset of the totalplaylist for a given show which includes the label or “bookmark” for thesegment currently playing.

[0314] The index, a vertical list of the program's segments displayed onthe left or right side of the video image, with the beginning segmentsat the top, provides an easily understood guide to the content of theprogram currently being viewed. When the segment index is displayed, thecurrently playing segment is highlighted as shown at 311 in FIG. 3 andat 411 in FIG. 4. The index can be easily taken off the screen with aclick of the remote (using the “segment guide” button to toggle theguide ON and OFF).

[0315] The index comprises a list of “slugs,” brief two to five wordlong descriptions of each segment. In general, the length of slugs willbe relatively short. When a high resolution screen is available, theviewer may be offered the option of displaying more detailed textdescriptions. Regardless of screen size, at times even shortdescriptions could trail off the screen, but the view may use the cursorbuttons on the remote to scroll the text of the index sideways.Alternatively, a wider horizontal space may be provided to display thedescriptive text for the currently selected segment (“selected” meaningeither the segment label (called a “slug”) that describes the segmentthat is playing or a different one being “pointed” to in response to theviewer's movement of the highlighting using the remote's UP and Downcursor buttons). This wider text for a selected label in the indexlisting could be the same as, or different from, the information shownin the information pane 412 for the highlighted segment as thehighlighting changes.

[0316] Example Show Playlists

[0317] Two example show playlists appear below, the first for a CNN newsbroadcast and the second for an episode of “ER.” In both cases, only thestart time, end time, the brief “slug” description and the longer“detail description” of each segment is shown, with additional attributedata omitted: EXAMPLE CNN Playlist Start Time End time Slug DetailDescription 00:10:000 02:49:000 Campaign Reform Campaign finance reformbill in Senate 02:52:000 03:19:000 McVeigh Ruling McVeigh requests notto be autopsied 03:22:000 03:49:000 Papa John's Papa John's vs. PizzaHut 03:51:000 04:53:000 Energy Crisis California faces rolling blackouts04:54:000 05:19:000 Mid-East Peace Israeli Minister meets with President05:20:000 05:44:000 US/Japan Meet Bush meets Japanese Prime Minister05:45:000 06:11:000 Submarine News Sailor to testify about accident06:12:000 08:17:000 Foot & Mouth Brit farmers balk at animal killings08:18:000 09:40:000 Weather Extended national forecast 09:41:00010:03:000 Buddah Destruction Ancient Buddhist statues destroyed10:04:000 10:34:000 Super Aspirin? Plavlix findings 10:35:000 10:58:000Space Station Mir continues in orbit, delayed fall 11:00:000 12:27:000Your Health Mediterranean diet 12:28:000 13:27:000 Commercial 1 OfficeDepot commercial 13:28:000 13:58:000 Commercial 2 Shell commercial14:00:000 14:48:000 Top Stories Power crisis, campaign reform, McVeigh14:49:000 15:08:000 Wall Street Dow, Nasdaq, S&P reports 15:09:00015:18:000 Dollars & Sense Waiting on federal reserve 15:19:000 16:01:000Interest Rates Federal Reserve rate cut analysis 16:02:000 16:23:000Cholesterol Drug Pfizer has competitor

[0318] EXAMPLE ER Playlist Start Time End time Slug Segment Description00:00:00:893 00:02:17:849 Opening Act Pigeons invade the ER 00:02:17:84900:03:07:757 Opening Credits Show theme and credits 00:03:07:75700:03:39:201 (C)Francesco Rinaldi The ultimate spaghetti sauce00:03:39:201 00:03:54:105 (C)JC Penny Go on a shopping spree!00:03:54:105 00:04:24:229 (C)Sprint PCS Special cell phone promotion00:04:24:229 00:04:39:659 (C)Maxwell House Good to the last drop00:04:39:659 00:05:09:060 (C)Mazda Sports Car 2001 Mazda Protégéfeatures 00:05:09:060 00:06:11:587 (C)NBC Promo Promotion for Dateline00:06:11:587 00:13:55:633 Act 1 Beauties hurt in salon explosion00:13:55:633 00:14:56:132 (C)Office Depot Office Depot 00:14:56:13200:15:11:086 (C)Milka Chocolate Milka - Little Scoops 00:15:11:08600:15:41:252 (C)Saturn 2001 Saturn SC1 00:15:41:252 00:15:57:176 (C)JCPenny Go on a shopping spree! 00:15:57:176 00:16:26:674 (C)CharlesSchwab IRA - starring golfers 00:16:26:674 00:16:56:117 (C)AT&T WirelessAT&T - Customer Advantage 00:16:56:375 00:17:06:107 (C)PGA Tour PGATour - Tiger Woods 00:17:06:107 00:17:38:048 (C)The Weakest Link Newgame show promotion 00:17:38:048 00:28:24:255 Act 2 Benson still can'tfind a job 00:28:24:255 00:28:54:126 (C)Honda Civic All new 2001 Civic00:28:54:126 00:29:08:904 (C)Crouching Tiger Crouching Tiger - HiddenDragon

[0319] In the display shown in FIG. 4, the video picture is shrunk toprovide an empty area for the index in on the side, resulting in extraspace in the vertical dimension that is used for a multiple-functioninformation pane 412. This space may be used for a variety of things,including displaying a more detailed description of the highlightedsegment, or for displaying advertising. The additional information aboutthe selected or playing segment may include a short synopsis or weblinks associated with the content as illustrated at 510 in FIG. 5.

[0320] Advertisements displayed in the information pane 412 may beselected based on the content of the then-selected segment. Forinstance, an advertisement could be displayed at 412 for a video ondemand (VOD) movie related to the topic in the then-playing segment.These ads could also be interactive, allowing a viewer to “click on” thead to play a recorded full-motion promotional video, or to link to a website using the data download facility 240, while the previously playingprogram was paused (that is, concurrently recorded for latertime-shifted viewing if desired). The advertising displayed in theinformation pane 412 may also be personalized to each household, or tothe current viewer, by using stored preference or demographic data toselect advertisements which are more likely to be of interest to theparticular household or viewer. A combination of content monitoring andviewer/household preference data may be used to select the advertisementdisplayed.

[0321] The index list of program segments may include one or moreadditional levels of nested subindexes. For example, the segment label“Ad break” at the bottom of the index shown in FIGS. 3 and 4 might bepresented as shown below:

[0322] Ad Break

[0323] Wheaties

[0324] Zoom Preview

[0325] Space Ranger

[0326] Ball Park Franks

[0327] The subindex, called a “twistie,” would offer viewers theopportunity to access a finer division of the show when desired, whilepreserving screen real estate when not needed. The twistie hierarchycould have two, three or more levels of detail. When the playback of theprogram arrives at the location in the video represented by a twistie,or when the viewer scrolls down to that point with the remote pointer,the twistie can automatically open up to show the detail about thecontent at that point in the program. Alternatively, the twistie couldopen up after the viewer inputs a specific signal with the remote to doso.

[0328] Independent of the length of each segment being shown, a visualdisplay might display the length of the entire program as a time bar asillustrated at 520 in FIG. 5, running either vertically or horizontallyacross the screen. This time bar could have markers indicating thelocations, and end markers, for each segment. An indicator on the timebar, such as the line 522 connected to the highlighted, currentlyplaying or selected slug “Ernie Sings,” would show viewers the locationof the currently playing or selected content. For playlists thatcontained only a subset of all segments, this view would be instructivein giving viewers a sense of how the show was compacted.

[0329] Attributes about a specific segment may be displayed in theinformation pane or in pop-up box located in a column beside therespective segment slug. Displayable attribute information could includethe length of the segment, the category into which it has beenclassified or other information regarding the subject matter containedin the segment such as its “rating,” or the name of the original programfrom which the segment came (in the case of a “metamix” of segments).

[0330] Other attributes might be more “analog” in their nature with theattribute column displaying the “level” of such qualities. For instance,the attribute column might have symbols representing the level of adultcontent in each segment (perhaps separate levels for violence, profanityand nudity), or an indication of segments' popularity as measured byfrequency of viewing by others (“Hotspots”). Database Description

[0331] Metadata which describes programs and the segments which make upthose programs may be advantageously stored in a relational,hierarchical or object-oriented database, preferably with the ability toaccept and deliver data in XML format.

[0332] The principal data entity stored describes an instance of a“show;” that is, an entity that defines content that is made availablein a specific market, at a specific time, on a specific day. Thisentity, called a ShowInstance, is a data object from which market data,general show data, and specific segment data can be extracted in orderto separate instances for reusability and scaling purposes.

[0333] Each ShowInstance specifies its airdate, what market it is in,what show it is, and an ordered list of GSpots (detailed informationabout a program segment) with start/end times. A Market will be definedby zip code, MSO, and name, as well as potential scheduling information.Generalized Show information consist of the show's name, type, duration,and general synopsis data. A key part of data will be our GSpots. TheGSpot will be our detailed breakdown of shows.

[0334] The ShowInstance contains an ordered set of GSpots whichconstitute a playlist for the ShowInstance. Each GSpot will containdetailed information on a segment; that is, on a truncated section ofvideo, type, caption, and rating information all specific to thattruncated section.

[0335] The database may further hold User Generated informationconsisting of User contributed GSpots, as well as potential membershipinformation. The user section of the database will grow as the user baseexpands and uses the system and provides a potential for the gatheringof demographic, viewing, and usage data that may be used to identifyindividual preference data for users as well as aggregate date such aprogram popularity ratings and “hotspots.”

[0336] The database may futher include Searchable Text data beyond thesegment slugs and describers already noted. This text data may includereviews, cast lists, narrative descriptions, close captioned text, andkeyword descriptors, all of which may be searched to enable the viewerto search for and identify programming of interest, programming thatshould be deleted., or programming to be bookmarked to form user-madeplaylists.

[0337] The viewer may be provided with the ability to “sort” or “select”(filter) program segments based on at least selected ones of theseattributes by clicking on the attribute label or icon in much the sameway a PC email client program user can sort email by date, sender, etc.by simply clicking at the top of the respective column of an emaillisting. The system may display any number of attributes at a time, andviewers will be able to rotate or scroll through different attributepresentations by clicking their remote.

[0338] The Index list visually highlights the segment currently playing,while a highlighter that is moved by using the up-down cursor keys onthe remote indicates the segment the viewer is “pointing to” or has“selected” on the list. If the viewer clicks the remote when a listedsegment other than the one currently playing is highlighted, thepointed-to segment becomes the segment that is playing.

[0339] If a viewer attempts to move the highlighting downwardly from thelast segment listed, or upwardly from the first segment listed, whilesearching for a new segment, the list can simply scroll or can be splitinto two windows, with one displaying the segment being viewed (andperhaps a few adjacent segments to give the viewer a sense of where thecurrently playing segment is within a show), while the other windowsshow lists of segments that are off the screen through which the viewermay wish want to scroll.

[0340] The viewer may also be shown a listing of the specific segmentswithin the index that have already been viewed (where “viewing” might bedefined as seeing at least 25% of the segment). Partially viewedsegments could be visually accented in another way to distinguish themfrom those that have been fully viewed. This feature would be similar tothat feature of email client programs that shows unread emails in boldtype while un-bolding the listing of messages once they are read.

[0341] The system may create a placeholder or temporary bookmark when aviewer leaves a segment. If the viewer returns to the segment, thisbookmark will behave like a new bookmark placed between the beginningand end of that segment. This feature will allow a viewer to surf out ofa segment to look for another segment to watch, a different recordedprogram to peruse, or simply to take a break from viewing. In any case,the viewer will be able to easily return to this point of departure uponreturning to the segment by clicking the next segment button. A visualsymbol on the index could indicate that a particular segment has abookmark in it. If the user watches the unviewed portion of the segment,the bookmark would be automatically deleted.

[0342] As the index will take up valuable screen space, the viewer willnot always want to have it visible. At the same time, the viewer wouldwant to have the ability to skip to the next segment without having totoggle on the index display. Therefore, the remote control shouldprovide a “Next Button” that will skip the playback to the next segmentwith a simple click without the presence on-screen of the index. Inaddition, a “Back” button will take the user to the previous segment.The “Next” and “Back” functions would respect the current sort order ofa playlist, e.g. a playlist sorted by alphabetical order would have C asthe next segment after B.

[0343] Holding either button down could have the same effect as it doeson the keyboard—resulting in an accelerating rate of skipping.Alternatively, such an action could simultaneously result in the indexbeing displayed again without having to have specifically hit the “viewindex” button.

[0344] Viewers may also able to invoke fast forward and backward whileplaying a segment. Thus, segment management enables a viewer to makegross movements in the video stream, while fast-forward and reverse moveyou through the segments in an analog way. A toggle button on the remotemay be provided to cycle the viewer through multiple fast-forward (orbackward) speeds and then back to normal speed.

[0345] The system may also provide a “scan” function that behaves likethe scan button on a car radio, which goes from station to stationplaying a few seconds of audio from each station until the listenermakes a choice. In this case, when invoked, the scan button will play afew seconds of video from each listed video segment, moving from choiceto choice until a selection is made. This scanning concept can work atboth the show-level and segment-level of granularity. Specifically,scanning can be used within a program recorded on a PVR or stored on aVOD server to help the viewer decide which segments are worth watching.The scan feature may also be used for a set of programs that are eitherstored on a VOD server or recorded to a PVR, scanning through samples ofeach recorded program to help a viewer select which program to watch.

[0346] For either the segment or show-level scanning function themetadata could be generated via:

[0347] 1. Human editors selecting the appropriate content (done inadvance of playback);

[0348] 2. Automatic markup techniques used instead of or to assist humaneditors, or at time of playback at the client or server system; or

[0349] 3. A random content selection means employed at the time ofplayback under the assumption that any piece of content could besatisfactory.

[0350] At the segment level, the scanning function would march down theshow index; showing a snippet for each segment and helping the viewerdecide if any given segment was worth watching in total. The techniquecould be used for any playlist generated for a given show or set ofshows (a “metamix” playlist). Segment scanning may be performed in oneof two ways:

[0351] 1) A viewer could hit the Scan button and hit a Play button whena segment worth watching was presented. Upon hitting the Play button,the segment would start at the beginning of the segment. Uponcompletion, the system would need to be reset into scan mode.

[0352] 2) The Scan button could take the viewer through each segment'ssnippet and the user could signal which segments were to be watched uponcompleting the scan. This would in essence, create a playlist ofsegments to watch. It would have the advantage of ascertaining which ofall the segments were worth watching in full before committing the timeto watch any one segment in its entirety.

[0353] At the program level, the Scan Function is akin to channelsurfing (where a viewer watches a small snippet of show before movingon), or alternatively, serves to replace analyzing the EPG to see whatis worth watching. When used at the PVR level, the scan function wouldselect a single scene, a few snippets, or an integrated “trailer” togive someone a sense of the program.

[0354] Typically, the metadata used for program scanning would usemetadata that was different than that used to divide the show intosegments, as the index metadata would be too course to allow for theextraction of the short, pithy highlights best suited for this function.

[0355] The metadata used in the scanning function, and the resultingvideo clips extracted, would have to be carefully selected so as not togive away too much of the plot or game result in the case of sports, asdiscussed later under topic “Plot Leakage.”

[0356] The content used in either program-level or segment-levelscanning could be customized to a specific viewer, client device, or VODsystem. This playback could be custom-tailored to each viewer's taste inthe following ways:

[0357] 1. Length of snippets

[0358] 2. Degree of “plot leakage” allowed

[0359] Type of content. For instance, if a viewer liked actionsequences, the system could use metadata that would play samples of theaction scenes for scanning purposes.

[0360] Playlists

[0361] An index listing of segments is manifested as a collection ofmetadata which defines an ordered set of program segments that may bereplayed in the order listed in the absence of some intervention by theviewer. Hence, these segment indexes are also termed “Playlists.”

[0362] In general, as noted earlier, segments have any number ofattributes associated with them. These attributes could be the basis forcreating playlists, that is, a listing of segments that represents asubset of all the segments or a subset of some existing subset ofsegments. A “show index” is a playlist of the segments that makeup abroadcast program or “show” with the segments being listed in the orderoriginally broadcast. A user-created playlist may identify segments thatmay or may not be in a different order than the original broadcast. Inaddition, the segments in different playlists may have different startand stop times and not be based on any particular segmentation schemefor breaking the show into segments.

[0363] Viewers will have the ability to select one or more segments fromone or more show indices and create a playlist for later playback. Theselection of the segments could be automatic (for instance if a searchwas performed requesting a list of segments on a specific topic) ormanual whereby a viewer would peruse the indices and their attributesfor one or more shows and select manually segments to be added to aplaylist.

[0364] The manual creation process could be assisted by automatedtechniques. For instance, a football game could be sorted by type ofplay, and then a user could manually select all the pass catches ofinterest to create an All-Star playlist. Each playlist could at any timebe reordered by the viewer, again on a manual basis, or by sorting onspecific attributes (such as date, channel, subject matter, etc.)

[0365] To support such manual creation of a list, the system supports acut-and-paste function. Adding this set of commands (or any other systemfeatures) is implemented by adding, or reusing buttons on the remote,adding voice control to the system, or adding more menu choiceson-screen.

[0366] A dynamic playlist can be discarded after using it or could bepermanently stored as a named playlist file for later viewing. Suchplaylist files can also be sent to other viewer households for use insimilar systems.

[0367] User-Generated Bookmarks

[0368] The system provides means by which viewers can create and deletetheir own bookmarks. With the appropriate input device, users can writethe text of the “slug” description for each bookmarked segment. Thesebookmarks could either standalone, be written to a separate playlist ofbookmarked segments, or they could be incorporated as extra bookmarks inan existing playlist. The bookmarks could be associated with a uniqueindividual within a particular household, either as a guide for a viewerto previously watched material, or as a recommended bookmark or playlistof bookmarks written for another viewer.

[0369] Viewers could also manufacture other attributes and associatethem with a segment. For instance, a viewer may wish to keep a playlistbased on Hotspots (see below) but modify later modify that playlist byincluding her favorite music video in the list. The system would allowher to insert a new Hotspot rating for that segment. By the same token,attributes such as category classifications for segments could bechanged.

[0370] Playlist Selection

[0371] The viewer interface should offer viewers an easy way to selectamongst playlists, perhaps without having to stop the show. Using abutton on the remote, a user could toggle between multiple playlists.The system would rotate through the various playlists showing them oneat a time to the user. The video would continue playing based on theoriginal playlist sequence unless the user actually selected a new one.If the user selected a new playlist, segments from the new playlistcould start playing immediately at the beginning of the newly selectedplaylist, or it could start playing off the new list at the same pointthe old one left off. The system could allow user-generated bookmarks tobe represented as a playlist and give users access to these via the sametoggle function.

[0372] Pre-Made Playlists

[0373] A “preview” playlist uses metadata to form a playlist which isthe equivalent of a personalized trailer for a stored program, linkingtogether segments of the show video in a seamless fashion. The previewauthor would take pains to be sure not to spoil the plot. The viewercould set parameters specifying such things as (a) how long previewsshould be, (b) from how many spots in the show the system could drawsamples, (c) how conservative the playlist should be in protecting theplot line, or (d) whether the viewer will accept random samples fromshows that weren't expressly tagged for this purpose.

[0374] Automatic preview construction may also take into account thecontent preferences of a viewer. If a viewer really likes actionsequences, the focus could be on showing the viewer what actionsequences were in a particular show. The system would be able to deduceviewer content preferences over time by monitoring segment-based andprogram-level viewing. Alternatively, the user could specify suchpreferences explicitly in their user profile.

[0375] Highlights

[0376] A “highlights” playlist would focus on summarizing a givenprogram to a shortened version consisting of the highlights of the show.As in the preview playlist, the system could personalize the playlist byviewer or household based on time preferences, and content preferences.As is the case in previews and “favorites” described below, the segmentboundaries could be shifted for each type of playlist constructed.

[0377] The easiest form of highlighting is that applied to segment-ablecontent. Here the system would tag the best plays in a sports program orfunniest jokes in a comedy routine. Using Highlights to shorten linearprogramming to allow viewers to see the material in less time, whilestill preserving the plot, would be more challenging for the playlistauthor.

[0378] To provide the viewer with a measure of interactiveparticipation, a visible signal may be displayed on the screen when anupcoming segment is about to be skipped. The user would be able tooverride the skip at that time or after the skip had occurred byclicking on an icon that would appear before and after the skip.

[0379] A text description that would scroll or be presented on thescreen in the manner of close-caption text could be used to summarizethe missed scenes. Another way to handle the skip would be to present amini-trailer of the deleted section, flashing mini-snippets to give theviewer the gist of what was missed. .

[0380] Favorites

[0381] A “favorites” playlist would collate all the segments regarding acertain theme or character. Thus, a viewer might wish to see all thescenes from a golf tournament involving Tiger Woods or all the segmentsrelated to a soap opera sub-plot.

[0382] Hotspots

[0383] A “hotspots” playlist would be based on the popularity attributeof each segment. The system would monitor a sample of viewers,preferably similar to the target user, and determine which segments weregarnering the greatest interest. This “level of interest” metric is asegment attribute that may be displayed on the screen and the viewercould sort on it.

[0384] Personalization by Content

[0385] The “preference” playlist presents a subset of content, such as ashow or a daily collection of shows, to the viewer based on the contentpreferences assigned to each viewer, PVR, set-top-box, or household, andthe categories assigned to each content segment.

[0386] Disk Space Conservation

[0387] The system periodically purges older program materials to provideroom for new programming. In general, previously stored programs areretained until more space is needed, and then these programs are deletedon a prioritized basis. Programming may be discarded at the request ofthe viewer, with a warning being issued if the program to be deletedincludes one or more segments specified on one or more playlists.Programming which has been viewed and which has not been identified by abookmark or playlist will be automatically deleted to make additionalroom. Programming which has not been watched, or tagged to be saved, isdiscarded on a first-in, first-out basis to make room for newprogramming.

[0388] The memory management mechanism keeps track of relationshipsbetween segments. So if one segment duplicates another, only one wouldbe retained, and all playlist references to a duplicate segment to bedeleted are replaced with a reference to the retained copy. Whenduplicate segments exist, the more up-to-date segment is retained andthe older duplicate is purged.

[0389] PVR viewers could specify that they only want to watch particularshows using certain playlists. In such cases, the system could deleterecorded content segments not pertinent to those playlists. Forinstance, if only baseball highlights were desired, then the hard drivecould delete all non-highlighted segments once that metadata wasdownloaded. This process could be applied to segments with low Hotspotratings, or those that did not meet content parameters. When the systemneeds additional space, it may request the user for permission to deleteall portions of certain programs except those segments that arereferenced on one or more playlists.

[0390] EPG Annotation

[0391] Many of the functions and features described here with respect tothe indexing and playback of segments within a program may be applied toprograms or “shows” as a whole. At the “show-level” of granularity,these features and functions may be implemented by modifying andextending the existing EPG mechanism provided by the PVR or programmingprovider.

[0392] A conventional EPG may be annotated in several ways to implementthese features and functions. First, the displayed program guide may beannotated to identify locally stored programs in the PVR (or remotefirst way would be to annotate the EPG in such a manner to disclosewhich stored slowly stored programs offered as VOD) for which playlistsor other metadata is available, and further to identify what kind ofplaylists or metadata available (e.g., hotspot playlists, highlightplaylists, personal preference playlists, etc.).

[0393] In addition to identifying stored programs for which metadata wasavailable, the EPG for future programming may be annotated to identifythe extent and nature of the metadata which would be available forindividual programs, thus potentially assisting viewers in theirselection of which programs to record. Note also that the availabilityof metadata may be used as one of the factors taken into account whenthe system automatically selects certain programs for recording whenempty space is available.

[0394] A second form of annotation would show summary data concerningthe metadata available for a show. Summary “hotspot” data, for instance,could be used to visually identify, for example by color coding, thoseprograms which were watched only briefly and shows which audiencecontinued to watch for extended periods. These viewing statistics couldbe compared to past episodes to get a relative measure of a given show'spopularity versus other episodes of the same show. Again, for a givenviewer, the goal would be to offer collaborative data from a group ofviewers whose characteristics matched those of the PVR user.

[0395] Another form of EPG annotation could be used to identifyprogramming which matched the viewer's content preferences. For example,the identification of a viewer's favorite actors could be stored aspreference data, and when one of the identified actors appears in aparticular show, that show could be highlighted on the EPG, and segmentsin which that actor appeared could be identified in a playlist. Forfaster-breaking items, an alert that new content of interest hasappeared could be displayed on a segment index listing, on theinformation pane, or on an EPG.

[0396] Note that all of these techniques can be applied on bothprogram-level EPGs and on the segment-level segment index listings.

[0397] Movie Rating Control

[0398] As noted above, attributes may be associated with each segmentidentified in a show index or other playlist. These attributes mayinclude a ranking for each segment by its level of sex, violence, andprofanity.

[0399] The advantage of using this technique is that it is entirelyunder the control of the user. The family editor may use automatedfiltering to create a playlist guided version of a show which deletesobjectionable material, and has the opportunity to review questionablesegments and could perhaps leave them if desired. To do this, the FamilyEditor would have to have the ability to change the rating of segment ordisable the offending attribute. This would be done via a simplethumbs-up/down type of one-click action on the remote. Furthermore, thefamily “suitability reviewer” could use the “Scan” mode to quickly getto the pertinent part of each segment to ascertain its suitability.Finally, the playlist as edited for a given show could be easily savedfor later reuse allowing the movie to be filtered at a different timefrom the playback.

[0400] To give viewers a sense of what they missed of plot-importance,if anything, the index listing could be used as a tool to inform. Thatis, certain segments could be deemed to be important from a plot orcharacterization standpoint, and the extended slugs or info-boxes forthese segments could reflect this information. The video screen would beblank or display a still frame, while viewers would take a few secondsto read about the missing piece.

[0401] Note that the use of an edited playlist for providing anexpurgated version of a show does not prevent others from watching theshow in its entirety. Thus, parents may filter movies for theirchildren, but may watch the same movies in their entirety when theirchildren are not present.

[0402] As an alternative, a visual flag may appear when a segment havinga defined attribute (such as high violence content) is about to appear,alerting a parent to the possible need to skip the segment manually ifit is deemed to be inappropriate for viewing. In this system, when anundesirable segment (as determined by the segment attribute and viewerinput as to what constitutes “bad”) is about to come on, a visual signalwould appear on the screen. There could be an indication as to how longuntil the undesirable scene was to appear (perhaps a count-down typetime bar) and perhaps the length of the adult scene as well.

[0403] To this end, different icons could be used to indicate for foullanguage, nudity, and violence. Images of monkeys with hands over eyesand/or ears could further help guide the parent as to the correct courseof action. These visual clues would give the parent time to avert thechild's attention before the scene appeared. The adult would continue tobe able to watch the entire sequence with minimal distraction fromhaving to monitoring the children's exposure to undesirable content.When the adult content had passed, the icon would leave the screen.

[0404] In another form of “Viewing with Parents,” the default could bethat an adult segment would automatically be skipped unless the systemwas told otherwise. Under this scenario, a visual cue would tell theadult that a scene was about to be skipped. The adult could read theindex information, or decide based on other circumstances to overridethe setting and have the segment play.

[0405] Self Made Movie Bookmarks

[0406] As described earlier, the system may include means forautomatically parsing or “bookmarking” a stored program into segments.The system would also allow users to create their own metadata. Thus, asnoted above, a parent could watch a movie first and bookmark out theundesirable scenes. In addition, a viewer could use the self-madebookmark feature to easily get back to favorite scenes. Such a featurewould be especially popular with children who like to view the samemovies over and over. Note that a user-created bookmark may take theform of additional metadata that appropriately “tags” an already parsedsegment in an existing playlist, or an item from an existing playlistmay be added to a new, user-created playlist of bookmarks. In addition,the bookmarked item may be parsed (that is, have its beginning andending times specified) from existing content.

[0407] A viewer may transfer playlists such as self-made sets ofbookmarks to other viewers, or upload them to a server using the uploadfacility 250 seen in FIG. 2 where other users could access them.Community-generated bookmarks would clearly be marked as such and bynoting the author, so that other users could get a sense of who produceduseable markups. Users would be able to rate the value of the markupsprovided by a given reviewer and provide attributes for the playlist,such as an indication of the playlist's suitability for children of aparticular age.

[0408] Plot Leakage, Dynamic Descriptors and Segment Markers

[0409] For information-based shows such as a news program, the labels(slugs and descriptive text) for each segment should be as informativeas possible in order to increase the efficiency of the viewer's surfingactions. When displaying the descriptions of segments for a sports eventor drama, a problem could develop by exposing the viewer to too muchinformation, giving away the plot, result of a game, etc. This “plotleakage” could be avoided by having an alternative set of segment labelsthat do not give away the story but still act to inform the viewer to alimited extent about the particular segment, much as the labels on DVDsegments do.

[0410] Consequently, the system supports labels whose content candynamically change as the show is watched. For instance, after a portionof a baseball game has been viewed, the labels for the already viewedinnings could be changed to be very descriptive. This change would bemade under the assumption that the viewer had already seen the contentonce and that they would only go back to review certain items a secondtime to see specific highlights or other points of interest. As such, itwould be important to label the previously viewed content as well aspossible, while keeping the to-be-viewed content labeled in aless-informative manner. The user may affirmatively select informativeor generic labels under different circumstances.

[0411] In addition, segment markers could change over time. Forinstance, after the ball game had been viewed through the first threeinnings, the segment markers for that material might focus on pointingout the highlights of those innings. The pointers for the material thathad not been viewed would demarcate the beginning and end of innings,for instance, or other generic boundaries that do not reveal the contentof the show.

[0412] The Vault

[0413] A major benefit of the playlist-based metadata employed by theinvention is the ability which is provided to viewers to save a show, orselected segments of a show, in semi-permanent form, and potentially ina manner that varies the saved program itself. This feature allows usersto store entire shows, or selected portions of shows, in their ownpersonal virtual library or jukebox for quick, personalized viewing.

[0414] A key ease-of-use feature for the Vault would be the “one-buttonsave”. When watching a show, a viewer could hit the Save Segment buttonand the segment currently being viewed would go into the vault (that is,metadata identifying the current segment would be added to storedplaylist).

[0415] Another feature would allow segments fitting a particular profileto be automatically saved to the Vault. For instance, a soccer fan mightset up his system to automatically save any recorded goals to his Vault.In a VOD system, this virtual saving would happen when the new contentwas added to the server and associated with the respective metadata.There would also be a reference as to the origin of the new material inthe Vault.

[0416] The use of playlist metadata to identify “saved” segmentsoperates as a “virtual vault,” conserving disk space in that copies ofthe recorded program segment are not normally made. If the memorymanagement mechanism attempts to delete a program containing segmentsidentified in a playlist, all portions of the program except thereferenced segments may be deleted. Alternatively, the memory managementsystem may automatically and transparently copy referenced segments intoa separate file and alter the playlist(s) to refer to the segments inthe new file, making the original program file or files eligible fordeletion. In the case of a VOD system, playlists for individual viewersmay be stored at the server providing the content, or may be savedlocally at the PVR, in which case the metadata includes references tosegments as stored on the VOD server.

[0417] Note that playlists may organized into personal “folders” whichmay be password protected so that they can be accessed only by theirowners, or playlists may be placed in a common area could be accessibleby anybody.

[0418] Further, playlists may be organized into content based folders,such as Sports, News, Comedy, Music, etc. that may be pre-defined orconstructed by the viewer. When a playlist (which may consist of only asingle segment) is saved, it would be assigned to one of these specificstorage areas. Note also that both folders can be nested, and the usermay request the sequential playback of the contents of a folder. Thus,folders may be treated as a playlist of nested playlists. By way ofexample, a viewer might request the playback of a folder of “musicvideos” which contains a folder of “Jazz” videos as well as playlistsfor individual music videos. The sorting, filtering and editing toolsdescribed above for playlists may be used to organize the content offolders, and derive new folders from existing folders.

[0419] The Vault may also support copying portions of programs topermanent storage, i.e. DVD or CDR. For instance, this feature couldsupport a user copying an MTV music video onto a DVD for a permanentcollection. Any metadata associated with the segments could betransferred to the DVD as well, particularly any segment markers as theycould allow the users to use the DVD controls to surf through thecontent on the DVD. Business rules, digital rights management, andpayment systems could be associated via metadata to the content allowingor disallowing such functions.

[0420] Preferential Segment Recording

[0421] Conserving disk space on the PVR or nPVR (network PVR) is aconcern to both users and MSOs. One way to conserve disk space is torecord only certain parts of a show by informing the PVR or nPVR aheadof time specifically what to record or what not to record. Analternative mechanism which achieves similar results by using metadatato trim the unwanted portions after a recording. The ‘trimming’ processmay be accomplished in at least two ways: by editing the recording todelete the unwanted portions leaving just the desired segments and bymaking a new copy of the recording but with just the wanted portion ofthe video and then delete the original. In both cases, the metadataassociated with the wanted segments would be used to identify theportions to be ‘trimmed.’ This process may be automatically invoked uponcompletion of recording or manually processed with the user selectingindividual segments to delete or keep. In addition, it could be possibleto have the system automatically ‘trim’ certain pieces of a recordingbased on a user's viewing history and personal preferences.

[0422] Business Rule Support

[0423] One or “business rules” may be associated with a folder, aplaylist or an individual segment within a playlist. For example, acontent provider may set up two or more classes of service, such as‘paid’ and ‘non-paid.’ The ‘paid’ subscriber would have access tometadata identifying advertising and be permitted to manually orautomatically watch commercial free TV. The ‘non-paid’ subscriber wouldnot have access to these marks and have no automated way to avoidcommercials. As a second example, a provider might offer playlists whichprovide condensed highlights of a sports event only to premiumsubscribers.

[0424] These implementations allow for the creation of one piece ofcontent that can be delivered as two separate services. In the aboveexample, NBC could create one episode of Friends which could then becould be viewed differently by two classes of service.

[0425] Digital Rights Management

[0426] Digital rights management is a concern to operators and contentowners alike. As users adopt DVR technologies to take more control oftheir own viewing habits, content owners need new ways to maintaincontrol of their content. By using metadata to specify business ruleattributes for folders, playlists and/or individual segments, differentrules can be applied to different content. For instance, some musicvideos recorded from MTV could be saved indefinitely while others mayhave specific expiration dates. Similarly, one video may have unlimitedplaying capability while another has a defined limit, e.g. after 10plays the user can opt to pay for additional viewing rights or let thecontent expire. Thus, each segment or group of segments can beassociated with attributes that specify the rights a user or class ofusers has with respect to that content.

[0427] Payment Systems

[0428] Business rule attributes provide a mechanism for providingadditional revenue opportunities to content providers by allowing usersto purchase special rights for a show or parts of a show. Similar to thesale of individual songs via the Internet, the system may permit a userto buy the right to play a copy of a music video or how-to video whichhas been recorded on the PVR. Moreover, a user could create their ownDVD or CD-R by buying the rights to copy to a permanent archive or apermanently saved portion of the PVR hard drive, a segment from atemporal PVR recording which was previously tagged for limited viewingonly. Information in the content's metadata, such as price or SKU, couldbe used to conduct the commerce transaction which could be conducted ata Web site having a URL identified in the metadata.

[0429] Metadata Distribution

[0430] The metadata download facility, seen at 240 in FIG. 2, like anyreal time data feed, is necessarily limited by network latency, networkbandwidth, and/or limited peak processing capability. Consequently,impairments in the reliability and timeliness of metadata could become acrippling issue for the service. By supporting two distribution methods,metadata delivery can be adjusted to match the actual end-to-endcapacities of the system thereby increasing the services' performanceand overall reliability.

[0431] The two methods of distribution may be used: rolling and batch.Using rolling distribution, each segment's metadata is transmitted uponcreation rather than waiting for the entire playlist to be completedbefore sending. Batch distribution occurs when transmission is delayeduntil a playlist is complete with all its tags, when it is transmittedby a post broadcast. Each component of the communications link, bothserver and client, are designed to support both distribution models. Thedecision to use the rolling or batch method can occur in both thetagging station and the back-end servers.

[0432] A configurable “blocking factor” parameter associated with therolling model specifies the the number of tags to group before sending.There are two primary factors that influence the value of thisparameter: the amount of time desired between tag creation and arrivalat the PVR, and the amount of bandwidth available for metadatatransmission. A very small blocking factor (such as one) would yield asubstantially constant flow of very small data packets, delivering closeto real-time metadata, and would have an extremely low total bandwidthrequirement. A large blocking factor would result in larger packetsizes, delivering in seconds to minutes, and have higher peak bandwidthrequirements. Because each MSO has unique network architecture issuesand different uses for the metadata, this parameter can be configuredseparately for each destination. In applications where the metadata isdelivered by a “dialup” connection, the batch method may be used tocollect playlists for transmission when periodic connections areestablished. The rolling model is much preferred, however, and workswell with data carousels commonly used in the cable operator's system.Because the data content of metadata is extremely small compared to thecontent, typically on the order of 5 kilobytes for each hour of content,the transmission of metadata does not add significant burden tobroadband system, and may be transmitted separately on a narrow bandsystem.

[0433] Note that the rolling distribution model may be hidden or exposedto the viewer. Using a configuration variable, the end device couldcollect rolling ‘tags’ and display a given playlist index to the useronly upon receiving the final ‘tag’ for a show indicating to the systemthat the playlist is complete, or may be displayed immediately when theuser is watching and recording live TV, providing a guide to the priorrecorded portion in the event the user wishes to replay portions evenbefore the program (and the associated playlist) have been completelyreceived.

[0434] Rule-Based Formatting

[0435] To cost effectively create metadata, playlists are created by themetadata service and then distributed via facilities provided by manyMSOs as indicated at 240 in FIG. 2. In general, this is an extremelyefficient way to create metadata; however, there may be circumstanceswhere slight modifications to the playlist(s) may be required to fit theneeds of individual distribution channels. In an NPVR (Network PersonalVideo Recorder) environment, for instance, each MSO may have differentrights to a particular program. One MSO could have the right to offer abaseball game with a complete, play-by-play index while another only hasthe right to basic navigation, i.e. inning-by-inning level of detail.Thus, although the content is common across the viewing universe, themetadata made available to different groups varies.

[0436] Since it would not be economical to produce different playlists(or sets of usable metadata) for each MSO using a human-operated taggingstation, a server-based set of rules could programmatically format theplaylist tailored to the rights or needs of the MSO. Accordingly, a‘master’ playlist, where the ‘master’ is considered the most complete,is produced first, and then customized playlists are generated from itby simply removing metadata that's not needed. As additional playlistsare distributed, they may be processed by the same script controlledconversion program from the master copy of each new playlist.

[0437] This approach has at least two key benefits: it becomeseconomically feasible to offer customized playlists to each destination,and it minimizes the amount of data that needs to be stored by onlyhaving to store a script for each MSO rather than multiple playlists foreach show.

[0438] Scalable Metadata Distribution

[0439] One of the major challenges in operating a real-time metadataservice of the type contemplated by the present invention is theefficient, scalable and timely distribution of the metadata itself. Withtens of thousands of cable head ends, and with each head end havingseveral different channel mappings, it is particularly desirable tominimize the amount of metadata transmitted.

[0440] To this end, the largest and most frequently distributedcollections of metadata are normalized prior to distribution. Playlistmetadata files are normalized to increase efficiency with the leastamount of complexity. This is accomplished by requiring each destinationor distribution node to have a small bit of intelligence which employs aconfiguration file to map the metadata to its local channelconfiguration. The configuration file distributed to end devices iscalled a Media Guide (MG). In addition to describing to a client devicehow to retrieve the metadata file (i.e. what the file name is), theMedia Guide also informs the client device which upcoming programs willhave associated metadata.

[0441] Using the Media Guide, a provider or client device may modify itsEPG to include indicators telling users ahead of time which programswill have metadata available for them. The Media Guide itself includescable head end identifiers and local channel numbers which, coupled withprogram broadcast times, are then mapped to a single unique identifierfor a program. This unique identifier is then used as the file name forthe metadata file containing normalized playlists for distribution.

[0442] Segment-Based Content Management

[0443] Video-on-demand (VOD) vendors today are looking at ways to usepeer-to-peer (P2P) networking technology and Akamai-like caching ofcontent to more efficiently push content to the edge of the digitalcable network. More popular offerings are cached at the system nodes asopposed to further upstream at headends (similar to the Akamai model) inorder to minimize bandwidth needs and latency. P2P techniques are beingdeveloped to allow headends and nodes to share content with each otherto minimize storage and bandwidth requirements in a manner similar tothe popular Napster music sharing system.

[0444] In accordance with one aspect of the present invention, asegment-based technology can bring a similar benefit to the cable VODworld. In this model, each show is segmented into logical pieces withsegment data coming from proprietary indexing services, contentproducers, the viewing community, or via automation tools of the typedescribed in the above-identified patent applications. With theavailability of segment breaks and labels, viewers will have the abilityto surf and peruse these video segments as they would text; that is,watching some full segments, some parts of others, and skipping somesegments entirely.

[0445] Data regarding this viewing behavior is garnered in several ways.Historical data can be used to deduce that program introductions areviewed the most, for instance. Human editors or the content producersthemselves could estimate segment usage after watching the contentthemselves. But better still, segment-based viewing data can beautomatically generated to indicate statistically how often each pieceis actually being viewed. Using actual data provides a model in whichthe usage data improves over time as more people view the content. Ofcourse, multiple models could be used with hypothetical data being usedat first and then being supplemented with empirical data as it becameavailable.

[0446] With segment viewing data, the most popular segments of shows canbe cached at the edge of the networks or made available for sharing byother nodes and hubs. In this way, bandwidth and storage resources, andlatency reduction, get optimized at the show or program level and not ona program-by-program basis.

[0447] As viewers gain experience with new capabilities (that is,watching highlights, previews, condensed versions of shows, etc.),viewing habits will change and content segments will start to fall intocategories that are seen often and those that are seen less frequently.Currently, without intelligent navigation to guide users to the partsthey really want, consumers use time-shifting navigation less then theywill in the future because the only means now available to most viewersis the fast-forward button. With the advent of easy-to-use time shiftingand program navigation controls, there will be less need for traditionalprogram guides because original broadcast times and channels matter lessand less. The new program guide will accordingly provide more and bettercontent information (content summaries, previews, shortened versions)and the guides will organize this information based on subject matterrather than by time and channel. The segmentation of programsfacilitates the delivery and organization of program content asdescribed in the above-identified applications and in the descriptionwhich follows.

[0448] To optimize this architecture, the system might not always storecontent segments exactly as defined by specific bookmarks. For instance,it might be beneficial if each section stored on the edge of the networkhad a few seconds of extra material from the next segment appended toit. That would allow viewers to finish one segment, begin the next, andskip to another without having the system have to present the wholeintermediate, but unwanted, segment that was skipped. If the viewers gotpast a “trigger point” in the appended material, that would signal thesystem to “download” the rest of the segment.

[0449] In an alternative embodiment, the edge network might store thebeginnings of each segment only. If the user viewed past a certainthreshold portion of any segment, the system would take that as a signthat the whole segment was to be watched and the rest of the segmentwould then be fetched.

[0450] To best implement such a system, means should be provided tosplice individual segments together in a seamless fashion. Existingvideo stream splicing techniques for this (some patented by ACTV) arecurrently used in inserting ads into video streams in a video stream,and could be employed for this new purpose.

[0451] Reducing Storage Requirements

[0452] If storage costs are relatively more expensive than bandwidthcosts, the system should be constructed in a way to minimize the numberof copies or variants stored for each program. One technique forreducing the total number of copies stored is to store more materialfurther upstream in the network from which a single copy can be sharedby more than one subscriber location.

[0453] By using metadata to identify and describe individual segments, asingle copy of the material to be used for multiple purposes, furtherminimizing the amount of redundant content in the network. For instance,a preview could be constructed from the existing material by usingmetadata to form a playlist of segments that could comprise a preview, ashortened version of the program, a version with the ratings changed(e.g. an alternative version with profanity eliminated), or a highlightversion that would just focus on one character.

[0454] The segment-based architecture provides an improved mechanism forpurging content from storage which is less likely to be requested soonagain. Currently, the PVRs manufactured by Tivo and ReplayTV use variousalgorithms for deleting complete programs based on age, perceivedutility to the viewer, and other conflicts. Using a segment-basedscheme, these algorithms could be more sophisticated. For instance, ifspace were needed, the first step might be to shrink down the program toa condensed version predefined by metadata. As a further example, if aviewer only watches MTV Hot Spots, then only these would be saved. Inother words, instead of just making a binary decision to delete or not,shades of gray could be offered by deleting part of a show. Anothermethod of “partially deleting a show” would be to re-encode in a lowerquality, or with a reduced picture size, or with color converted toblack and white.

[0455] Increased Processing Performance

[0456] Segment-based viewing can significantly increase processorperformance. In today's VOD systems, trick play (fast forward, rewind,and pause) is offered and used quite often. This trick play consumessignificant processing power. Each time it is invoked, the system has tostart playing from a sped-up file constructed for such purpose, orconstruct one on the fly by pulling I-frames out of the MPEG stream.Further computational efforts must be made to make the transition backto the original file for normal play once the new location for playbackhas been identified.

[0457] With segment-bases viewing, this processor load is lightenedappreciably as viewers can easily be moved across the original videofile. With a segmented file, users can get from segment to segment inone of two ways: by reading the index of segments that is presented andchoosing to jump to one, or by starting to view the next segment,deciding not to watch it, and then hitting the “next” button, bringingup the next segment in the playlist currently playing.

[0458] With segment-based viewing using these modes of skippingsegments, there will be much less use of trick play as a form ofnavigation. Viewers will merely click to get to the next segment orplace to sample content. With segment markers, each “stop” in theplaylist made is positioned to better delimit programming based oncontent, so that fewer but more intelligent moves are made.

[0459] “Ways to Watch” Television in VOD and PVR Systems

[0460] As described here, and in the related patent applicationsidentified above, a number of new “Ways to Watch” television aredescribed, playlist modes such as Previews, Hot Spots, CondensedVersion, Highlights, Personalized Versions, etc. These Ways to Watch canbe used to advantage in both VOD and PVR systems.

[0461] In subscription-based VOD services, these Ways to Watch may beoffered selectively to subscribers as extra-cost options. A secondbusiness-model concept is to use Previews (auto-generated or based onHot Spot real time ratings of segments) to induce people to buy specificshows.

[0462] These new Ways to Watch are preferably implemented usingsegmentation. It is important to note that a segment may be differentfor different playlist types. For instance, constructing a CondensedVersion playlist benefits from first identifying new, smaller segmentsthat are better suited to construct a summary. In a similar fashion,even smaller segments might be needed to make a Preview. Thus, for bothVOD and PVR systems, the playlist metadata used to specify the segmentsmaking up different presentations may refer to same program contentsegmented in different ways.

[0463] One of the useful new Ways to Watch television, “Hot Spots,” ispreferably implemented using playlist metadata to identify segmentswhich are deemed to be “popular.” Here it should be noted that there aretwo ways to think about them—as measures of popularity relative to othersegments within a show, or a more absolute measure which looks at asegment's popularity relative to segments in all shows or a group ofshows. To measure the relative “intra-show” popularity of a segment, forinstance, actual program viewing data would be processed to identifythose segments which were most frequently watched. Heuristics may beconstructed for each measurement to properly weight partial viewings.Users should be given the option of using either intra-show Hot Spot orglobal Hot Spot measurements.

[0464] Since Hot Spots will be found most useful when they are based onactual viewing data from a group of viewers with whom the subscribershares certain viewing habits and interests, the process of Hot Spotselection may advantageously employ collaborative filtering.

[0465] It should further be noted that, in assessing the popularity of agiven segment, it is important to take into account the order in whichthat is watched. Thus, if there were a group of like-minded viewer withwhom a subscriber shares bookmarks and Hot Spots, the subscriber mightalso want to share the order in which you viewed these segments as well.For example, a subscriber might send a playlist of music videos towatch. If the recipient merely watched the identified Hot Spots in somearbitrary order (such as the order in which the segments were recorded),the experience which the sender intended would not be reproduced

[0466] Presence and Interacting with Other Viewers

[0467] In addition to Hot Spot metadata created from monitored viewinghabit and explicit ratings submitted on segments, a mechanism may beemployed to provide “awareness-based communication” in the context ofparticular programs. The system tells a viewer “where the crowds are”and a viewer can then use that information to preferentially access theidentified content. This effectively creates a feedback effect in that,to the extent that more people flock to a show, the more people willwatch the show. Thus, the Hot Spot mechanism could indicate to a viewerhow many people (perhaps within a given circle of friends andacquaintances) are watching what segment of a given show at any time.This mechanism might could even allow someone to catch up to the crowdor to a friend. That is, someone could use the metadata to skim througha show to catch up to the portion being watched by a friend or a group,and then engage in a live chat with another person about the show atthat point. In this way, once a subscriber “joins the crowd,” she caninteract with those fellow viewers now understanding which segment theyare viewing. This could be done by visually associating each viewer witha segment in the index.

[0468] Because multiple people may be watching the same show atdifferent times, it would be advantageous to offer time-shifted orasynchronous communication between viewers. That is, comments, ratings,and other communications about a show could be saved and time-shifted.In addition, they could be sorted by associated segment.

[0469] The communication could be with known parties such as friends andrelatives, or with strangers. For friends, these comments could beorganized by originator and accessed as such. The comments could bestored at the headend, or for greater privacy, they could be stored onthe originator's machine. In the latter case, other viewers would onlybe notified of where to go to get the metadata. For strangers, amechanism maybe be provided whereby viewers would rate the commentators.The comments from all previous episodes would be saved for later access.As time on, users of the metadata would rate it and sort it out so thebest metadata rose to the surface for greatest use. In either model,viewers may review the metadata first, and use that as the “linkage” toaccess the content itself. That is they might find a glowing reviewabout a piece of content and click on the appropriate link to get thecontent itself, as discussed further below.

[0470] In another form of presence-based communication, the system couldallow a viewer to “pull a viewer” into a viewing experience by “InstantMessaging” the person. This communication could perhaps be based on aprofile or preference and key off of the context of where you were inthe show—i.e., a use of segment-based metadata. Historical data withregards to who watched the show last time could be used to “corral” theright people this time into seeing the show you have pulled down fromthe VOD server.

[0471] To facilitate finding out who to communicate with (amongfriends), the subscriber should be able to inspect into their viewingfile to see if they have seen the show before. Users could also shareviewing histories to request information about specific episodes. In PVRsystems, this mechanism may be extended to cover being able to preview,and perhaps modify, what they are about to record.

[0472] Metamixing: Inter-Show Segment Compilations

[0473] The segment-based system contemplated by the invention permitsusers to mix content across different shows creating in essence new“virtual shows”. Content could be pulled from any number ofsources—different VOD systems, a PVR, or off the Internet. The processof combining the content pieces would be done using supplied metadata.For instance, a user can select attributes (sport, team, company,character, actor, artist/singer, category) and have the systemautomatically and dynamically assemble content for viewing.

[0474] Additionally, if the VOD system knows what is stored locally,regionally, or nationally and its “availability”, the system can filterfor the viewer currently available or efficiently accessible content.

[0475] The metadata should describe the segments in enough detail suchthat redundancies could be avoided. In addition, the user or the systemwould need to provide bounds on the length of the composite showdesired. The composite itself would have bookmarks, and be surfable, andcontain information indicating the popularity of specific segments.

[0476] Ads Personalized by Segment Content

[0477] Based on the segment currently being viewed, advertising segmentscould be dynamically pulled from a database that would be personalizedby the type of content that surrounded the ad as defined by the metadataschema. Currently ads for a normal broadcast might be inserted based onthe surrounding contents (for example, advertising for snow tiresinserted in the weather report). In addition, advertising may beinserted into the viewing stream based on the personal traits of theviewer. As contemplated by the present invention, ad placement can be afunction of both the surrounding context and the identity of the viewer.

[0478] Automated Reviews

[0479] When a viewer has paused a movie and wants to pick up back wherehe or she left off at a later time, the system may automaticallygenerate highlights of the previously viewed segments to get the viewerback up to speed with the content. The viewer can choose to skip overthe review, or easily surf through it, if desired and in either casecontinue to watch the program from the point from which they originallyleft off.

[0480] Note, as mentioned above, the segments presented that wouldconstitute the review are not necessarily the same segments that wereused for other purposes and playlist. It would be ideal to use segmentsspecifically tailored to the idea of reminding people what they hadalready seen. (In the case of Previews, for instance, those segments arespecifically designed so as to not give away the plot, which is theopposite of the effect desired from a condensed review which is to serveas a reminder.

[0481] Automated Teasers

[0482] When a user has finished viewing a program the system willautomatically compile and play a list of segments associated withhighlights from programs that relate to the one just viewed. Theintention would be to grab the user's attention and induce him or her toinvest more time in the same type of content while they are still in theviewing process. This is similar to broadcasters' “coming up next week”gambit.

[0483] In the same vein, after viewing a show, the system may prompt theuser to “Recommend this show to a friend” by clicking on a button.

[0484] Sharing Content

[0485] Segment-based VOD systems offer some unique opportunities forcontrolling content sharing. For instance, content owners could assentto certain segments being shared with viewers outside the subscriberuniverse. This would allow someone to email a segment to another personwho is not a subscriber with the permission of the content holder. Thisform of sharing could benefit the content holder as it would stimulatedemand for the whole work in the same way that a radio station's play ofa single song stimulates sales of the entire CD.

[0486] Other business rules could be applied such as a limit on how manydifferent segments a viewer could send, which segments are send-able, alimit on how many could be sent to a given recipient, or how long ormany times the segments could be viewed once received, etc.

[0487] As with the sharing of bookmarks, discussed below, the sharingcan be done over an email system tied to the VOD system, via InstantMessaging, or other similar forms of communication.

[0488] Sharing Bookmarks

[0489] Within a VOD community of subscribers to the same content,segment metadata offers the opportunity to share bookmarks in a mannersimilar to the way in which PVR users share bookmarks. In a VOD system,however, it is no longer is hit or miss as to whether the recipient hasthe content to which the bookmarks apply—as everybody will have accessto the same VOD content. (In the PVR case, both parties must record thesame content for the sharing to be effective.) In a situation whereeverybody has access to the same content, the utility of receivingbookmarks from a trusted source is much higher.

[0490] Shared bookmarks can be either generated by the system or theycan be bookmarks generated by a viewer. The bookmark would have a “link”to the content, so that merely clicking on the link would “bring up” theappropriate program to view.

[0491] Library of Metadata and Bookmarks

[0492] Viewers could compile their own a library of bookmarks andmetadata. This can consist of user-generated metadata, system-generatedmetadata, or metadata received from other viewers. Users can add anddelete entries as desired.

[0493] The metadata could be sorted and viewed in various fashions.“Links” may be provided to take the viewer from the metadata to thecontent stored on the VOD system.

[0494] Whenever a viewer accesses content in the VOD system, it mayautomatically check the user's library of metadata to see if there areany self-generated or received bookmarks with which the content could besynchronized.

[0495] An alternative version of the library of metadata would be tohave the system provide a library of metadata that would function as anEPG. This metadata would consist of subjective metadata such ascommentaries, reviews, Hot Spots, as well, factual data such as segmentlabels, etc. Viewers could access the metadata and use the built-inlinks to access the actual content. Alternatively, this metadata couldbe combined with a normal EPG to get the best of both worlds.

[0496] The Bookmark Library as a “Previously-Viewed” EPG

[0497] While users will be able to create their own bookmarks, they willalso be able annotate the entire show in various ways. These annotationscould include comments about a show (if a keyboard or voice input systemis available) or simple ratings. This show-based metadata, as well as,any segment-based metadata, would be stored in a user-accessibledatabase.

[0498] The purpose of this database would be to maintain a running logof shows the user has viewed—perhaps both good and bad. With thisdatabase easily accessible, a user could access their history of viewingby title, rating, metadata information, channel, date, or any otherfield in the database.

[0499] To simplify the database, items could be deleted, either byalgorithm (“don't store newscasts in the database”) or manually, a stepthat could convert the database into a collection of favorites watched.

[0500] Segments or shows can be accessed in this manner. Once a show hasbeen identified, the viewer could click on it and invoke the relatedcontent. In this way, the library of bookmarks and database of metadatabecomes an EPG to previously viewed content. This would be handy whentrying to re-watch a previously viewed show, to just review thehighlights of such a show, to point out segments to another viewer youare watching TV with, or to retrieve metadata for sharing.

[0501] Personal VOD Vaults

[0502] As described earlier, a PVR may operate as a “vault” in which theuser can save segments from shows. In much the same way, a mechanism canbe provided to enable users to “save” segments stored on the headend PVR(Networked PVR). VOD headend storage is subtlety different from PVRstorage. The PVR Personal Vault is more focused around segments andrequires the user to purposefully put things in it. The VOD headendpersonal storage may be thought of as a Previously Viewed EPG, that is,a viewing diary that is somewhat more show-based, and would notnecessarily need a segmentation schema to work.

[0503] In this model, a viewer can choose to ‘save’ segments, eitherpre-defined or delineated by user-generated bookmarks, which have theeffect of storing the content in their own “personal storage locker” atthe headend. This storage really represents a shared right to thecontent stored on the cable company's PVR.

[0504] These rights could vary. For instance, they could be for apre-defined period of time based or number of viewings. There could belimits based on how much could be saved and what can be saved given theviewer's subscription level. Specific purchases might be required toview specific pieces of content. In other cases, “saved” segments mightbe able to be shared with others. This sharing could either be done byexporting the content itself outside the VOD system, or merely bysending someone the bookmarks describing the content's location (in muchthe same way that either a URL can be sent or an actual web page whensharing web content).

[0505] The act of storing also makes the content available in specificways, i.e., through the playlist manager of the Gotuit TV System.

[0506] The VOD vendor can store copies of the material in a way thatoptimizes bandwidth and storage costs based on the rights people mighthave to any piece of content. In one example, only a few segments out ofa show might be saved. The different types of vaults can be private,regional, or community-based and integrated with the VOD serverconditional access system.

[0507] Conclusion

[0508] It is to be understood that the specific methods and apparatusthat have been described are merely illustrative applications of theprinciples of the present invention. Numerous modifications may be madeto the processes and mechanisms described without departing from thetrue spirit and scope of the invention.

What is claimed is:
 1. A method for selectively reproducing recordedvideo program segments comprising the steps of: storing said videoprogram segments in a mass storage device, storing playlist metadataspecifying a selected set of said stored segments and the orderedsequence in which the segments in said set are to be reproduced on adisplay device in the absence of an intervening control command from theviewer, said playlist metadata further including a text description ofeach segment in said set, reproducing said set of stored segments onsaid display device in the ordered sequence specified by said playlist,at the request of a viewer, displaying a segment guide listing on saiddisplay device, said guide listing containing the text description ofeach segment in said set with the text description of the segmentcurrently being reproduced being visually identified on said guidelisting, visually identifying a different selected one of said textdescriptions on said guide listing in response to a selection commandfrom a viewer, and reproducing the video segment described by saidselected one of said text descriptions in response to further commandfrom said viewer.